Head impulse, nystagmus, and test of skew examination for diagnosing central causes of acute vestibular syndrome

Objectives

This is a protocol for a Cochrane Review (diagnostic). The objectives are as follows:

The primary aim of this review will be to assess the diagnostic accuracy of the HINTS examination for identifying a central etiology for AVS.

Secondary objectives

Secondary objectives include assessing the diagnostic accuracy of HINTS amongst the following subgroups:

• specific providers (e.g. physicians, non‐physicians)

• time from symptom onset to presentation (e.g. less than 24 hours, greater than 24 hours)

• reference standard (e.g. advanced imaging, discharge diagnosis)

• underlying etiology (e.g. ischemic stroke, alternative etiologies [hemorrhagic stroke, intracranial mass])

• study setting (e.g. outpatient [outpatient clinic, urgent care clinic, ED], hospitalized patients)

• individual diagnostic accuracy of each subcomponent of the examination (e.g. head impulse, direction‐changing nystagmus, test of skew)

Background

Target condition being diagnosed

Dizziness and vertigo are common reasons for patients to present for medical care, and account for approximately 3% of emergency department (ED) visits and 6% of general medicine clinic visits (Kroenke 1998; Newman‐Toker 2008a). This corresponds to over 3.9 million ED visits per year, with an annual cost exceeding USD 4 billion in the USA (Saber Tehrani 2013).

Acute vestibular syndrome (AVS) is a subset of dizziness and vertigo consisting of a constellation of symptoms, which can include severe vertigo, nausea and vomiting, spontaneous nystagmus (involuntary rhythmic side‐to‐side, up‐and‐down or circular movement of the eyes), and postural instability/imbalance (Hotson 1998). The etiology of AVS can include peripheral causes (e.g. vestibular neuronitis, neurolabrynthitis, Meniere's syndrome) or central causes (e.g. ischemic stroke, intracerebral hemorrhage, intracranial mass, bulbar ischemia) (Hotson 1998). Most patients with AVS will have a benign peripheral etiology (i.e. a non‐dangerous cause which can often be managed symptomatically) (Hotson 1998). However, dangerous central causes, such as cerebellar infarction and cerebellar hemorrhage, can mimic peripheral causes. Studies report that up to 25% of patients presenting to an ED with AVS may have a cerebellar infarction (Cnyrim 2008; Duncan 1975; Kerber 2006; Lee 2006; Norrving 1995). These central causes are more time‐sensitive, often requiring admission, serial neurologic assessments, and additional acute evaluation and management depending upon the specific cause. For example, an acute stroke may necessitate thrombolysis if the patient presents within 3 to 4.5 hours of symptom onset.

It is critically important to identify patients with cerebellar infarction because it can produce significant cerebellar edema (swelling) leading to compression of the brainstem and death without neurosurgical intervention (Hornig 1994; Hotson 1998; Sypert 1975). Additionally, patients with cerebellar infarction are at increased risk of recurrent stroke, so it is also important to diagnose this in order to reduce the probability of recurrent stroke (Amarenco 1994; Duncan 1975; Hotson 1998; Norrving 1995).

Index test(s)

The head impulse, nystagmus, test of skew (HINTS) examination consists of three individual exam components: head impulse, nystagmus, and test of skew. The head impulse test is a measure of the vestibulo‐ocular reflex (VOR) and is performed by having the patient focus their eyes on a single central target, while their head is rapidly rotated side‐to‐side in a horizontal direction (Halmagyi 1988; Newman‐Toker 2008b). A normal VOR response for this test is for the patient to generate an equal and opposite eye movement, such that their eyes appear stationary in space while their head moves (Black 2005). An abnormal VOR response occurs when the eyes are unable to maintain the fixation in one direction due to loss of the vestibular afferent input, causing the patient to develop a corrective gaze shift (i.e. their eyes must correct back to the center instead of remaining stationary) (Newman‐Toker 2008b).

Nystagmus (i.e. rapid, involuntary eye movements in a horizontal, vertical, or rotatory direction) is often present in patients with AVS from both a central and peripheral etiology (Newman‐Toker 2008b). When AVS is due to a peripheral etiology, the nystagmus is predominately horizontal and beats only in a single direction with the intensity increasing when the patient looks in the direction of the nystagmus fast phase (Cnyrim 2008; Kattah 2009; Newman‐Toker 2009). An abnormal finding (suggestive of a central cause of AVS) would include vertical nystagmus, torsion nystagmus, or nystagmus which changes in direction on eccentric gaze (i.e. does not localize in a single direction) (Cnyrim 2008; Kattah 2009; Newman‐Toker 2009).

Skew deviation consists of a vertical misalignment of the eyes due to an imbalance of vestibular tone in the oculomotor system (Brodsky 2006). This is typically assessed by performing an alternate cover test, wherein the clinician covers each eye in isolation, assessing for vertical correction of the eye position (Kattah 2009). A vertical correction would be abnormal and suggestive of a central cause (Cnyrim 2008).

A 'positive HINTS' examination is suggestive of a central cause and occurs when there is a positive head impulse test (i.e. normal VOR), direction‐changing nystagmus, or vertical correction on test of skew. This can be recalled using the mnemonic INFARCT (Impulse Normal, Fast‐phase Alternating, or Refixation on Cover Test). In contrast, a 'negative HINTS' examination is suggestive of a peripheral etiology and requires that the patient have an abnormal head impulse test, unilateral nystagmus, and no evidence of vertical correction on test of skew. The HINTS examination was first described in 2009 and, since then, a number of studies have evaluated the accuracy of this test (Kattah 2009). In fact, some research has even suggested that the HINTS examination may even outperform early magnetic resonance imaging (MRI) (Kattah 2009).

Clinical pathway

The traditional approach to patients presenting with AVS begins with a thorough history and physical examination. Depending upon the findings and the patient's risk of a central cause, clinicians may obtain additional blood testing, an electrocardiogram, or advanced imaging (e.g. ultrasound, computed tomography (CT), MRI). The approach to testing can vary with some institutions performing routine MRI in all patients with AVS, while others may only perform MRI among those deemed higher risk using the aforementioned criteria, which can include the HINTS examination. However, when compared with control patients, patients with AVS are generally more likely to receive additional testing and hospital admission, with overall admission rates ranging from 19% to 46% (Chase 2014; Dallara 1994; Newman‐Toker 2008a).

Alternative test(s)

Studies have demonstrated that individual historical and physical examination features are poorly predictive in isolation. Traditional risk factors, such as hypertension, diabetes mellitus, or history of ischemic stroke have minimal impact on the post‐test probability of a central cause (Chase 2014; Chen 2018; Machner 2020). Moreover, the description of the symptoms and chronicity have limited diagnostic utility (Chase 2014; Machner 2020), while the physical examination findings are more variable (Chase 2014; Chen 2018; Machner 2020). This has prompted many to rely more heavily on imaging.

The imaging test of choice for cerebellar and bulbar infarction is MRI with diffusion‐weighted imaging (DWI). While CT is often ordered, studies have demonstrated that the diagnostic accuracy of CT is substantially lower than MRI when assessing for ischemic stroke (Chalela 2007). This is even more limited when evaluating the posterior fossa, where CT is known to perform worse due to the bone‐related artifacts (Simmons 1986).

Despite the overall increased accuracy compared with CT, MRI has reduced accuracy within the first 48 hours. In fact, studies have reported false negative rates of 5% to 12% among early MRI compared to MRI obtained greater than 48 hours after symptom onset (Kattah 2009; Oppenheim 2000).

Rationale

While most patients with AVS have a benign etiology, it is important to accurately identify the more dangerous central causes. However, individual historical and physical examination features are of limited diagnostic utility, leading to increased use of resources and greater likelihood of hospitalization (Dallara 1994; Newman‐Toker 2008a). Therefore, there is a need to identify an accurate tool for detecting and excluding central causes of AVS.

Objectives

The primary aim of this review will be to assess the diagnostic accuracy of the HINTS examination for identifying a central etiology for AVS.

Secondary objectives

Secondary objectives include assessing the diagnostic accuracy of HINTS amongst the following subgroups:

• specific providers (e.g. physicians, non‐physicians)

• time from symptom onset to presentation (e.g. less than 24 hours, greater than 24 hours)

• reference standard (e.g. advanced imaging, discharge diagnosis)

• underlying etiology (e.g. ischemic stroke, alternative etiologies [hemorrhagic stroke, intracranial mass])

• study setting (e.g. outpatient [outpatient clinic, urgent care clinic, ED], hospitalized patients)

• individual diagnostic accuracy of each subcomponent of the examination (e.g. head impulse, direction‐changing nystagmus, test of skew)

Methods

Criteria for considering studies for this review

Types of studies

We will consider all primary diagnostic test accuracy studies if they evaluate the HINTS test used in a primary care clinic, an urgent care clinic, the ED, or during inpatient hospitalization against a final diagnosis of a central etiology of AVS as defined by the reference standard of advanced imaging or final diagnosis by a neurologist and that provide sufficient data to create the 2 × 2 tables to calculate sensitivity, specificity, and negative and positive predictive values. A central etiology will comprise causes involving the central nervous system (e.g. ischemic stroke, hemorrhagic stroke, intracranial mass). We will include prospective and retrospective studies, provided that the HINTS test was applied directly to patients. We will exclude retrospective studies where the tool was applied to records as opposed to actual patients. We will also exclude case‐control studies, case series, and case reports.

Participants

We will include adults (age ≥ 16 years) with symptoms of AVS who are evaluated in a primary care clinic, an urgent care clinic, the ED, or during inpatient hospitalization. If studies provide data on a mixed population (i.e. pediatric and adult patients), we will include the adult population if the adult data are presented separately. We will not include studies with mixed populations where adult‐only cohort data are not available.

Index tests

The index test will be the HINTS exam, which comprises head impulse, nystagmus, and test of skew deviation. We will consider any of the following findings as evidence of a central etiology: normal head impulse test, direction‐changing horizontal nystagmus, or skew deviation. The index test could be administered by a paramedic, emergency medicine technician, nurse, advanced practice provider, or physician (regardless of specialty). While we will collect data on the diagnostic accuracy of individual components of the HINTS examination when available, we will not include studies that only assess a single component in the absence of the full HINTS examination.

Target conditions

The target condition will be a central etiology for the AVS symptoms, which will be defined as any causes involving the central nervous system (e.g. ischemic stroke, hemorrhagic stroke, intracranial mass).

Reference standards

The reference standard for ischemic stroke, hemorrhagic stroke, or intracranial mass can include advanced imaging (e.g. MRI, positron emission tomography) or final diagnosis by a neurologist.

Search methods for identification of studies

Electronic searches

The Cochrane Stroke Review Group's Information Specialist will search the following electronic databases for retrospective and prospective studies of the HINTS examination. There will be no restrictions on language or year of publication.

• Cochrane Central Register of Controlled Trials (CENTRAL) in the Cochrane Library (latest issue)

• Cochrane Database of Systematic Reviews in the Cochrane Library (latest issue)

• MEDLINE Ovid (1946 to present)

• Embase Ovid (1980 to present)

The strategy that will be used to search MEDLINE can be found in Appendix 1. The Information Specialist will amend this where necessary to search the other databases listed.

Searching other resources

We will search ClinicalTrials.gov (www.clinicaltrials.gov) and the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) (apps.who.int/trialsearch/) for ongoing trials. We will also search the International HTA Database (database.inahta.org/) for published international health technology assessments. In addition, we will check reference lists of reviews and retrieved articles for additional studies, and perform citation searches on key articles. We will also search the grey literature using the CADTH Grey Matters tool (www.cadth.ca/grey-matters-practical-tool-searching-health-related-grey-literature-0) and Google via the 2Dsearch structured searching platform (www.2dsearch.com/). We will contact study authors for additional information where necessary.

Data collection and analysis

Selection of studies

Two review authors (MG, JNC) will independently determine eligibility of each study identified by the search. They will exclude studies that clearly do not satisfy the inclusion criteria, and obtain full copies of the remaining studies. Two review authors (MG, JNC) will read these studies independently to select relevant studies and, in the event of disagreement, a third review author will adjudicate (GDP). We will not anonymize the studies in any way before assessment. We will include a PRISMA flowchart in the full review, which will show the status of identified studies (Page 2021). We will include studies in the review irrespective of whether they report measured outcome data in a 'usable' way.

Data extraction and management

Two review authors (MG, JNC) will independently extract data using a standard piloted form and check for agreement before entry into Review Manager (Review Manager 2020). In the event of disagreement, a third review author will adjudicate (GDP). We will collate multiple reports of the same study, so that each study rather than each report is the unit of interest in the review. We will collect characteristics of the included studies in sufficient detail to populate a table of 'Characteristics of included studies' in the full review. We will extract the following information.

Study characteristics

• Study date

• Study design

• Study setting

• Study country

• Study duration

• Provider type and specialty

• Details of blinding and allocation concealment

• Follow‐up duration

• Publication type

• Study funding source

• Study author conflicts of interest

Participants

• Total number of participants

• Inclusion criteria

• Exclusion criteria

• Mean or median age

• Gender distribution

• Existing comorbidities

• Time from symptom onset to presentation

Index test

• Number of positive findings for the head impulse test component

• Number of positive findings for the direction‐changing nystagmus component

• Number of positive findings for the test of skew component

• Number of total 'positive HINTS' examinations

Outcomes

• True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN) results for the HINTS examination

• TP, TN, FP, and FN results for each individual component of the HINTS examination

• TP, TN, FP, and FN results for the HINTS examination when performed by physicians only and when performed by non‐physicians only

• TP, TN, FP, and FN results for the HINTS examination when performed less than 24 hours after symptom onset versus greater than 24 hours after symptom onset

• TP, TN, FP, and FN results for the HINTS examination when the reference standard is advanced imaging versus discharge diagnosis

• TP, TN, FP, and FN results for the HINTS examination when the etiology is ischemic stroke versus alternative etiologies

• TP, TN, FP, and FN results for the HINTS examination when performed in the hospital setting versus the outpatient setting

Analysis

• Statistical techniques used

• Subgroup analyses

• Number and percentage lost to follow‐up

Assessment of methodological quality

Two review authors (MG, JNC) will independently assess methodological quality using the Quality Assessment of Diagnostic Accuracy Studies version 2 (QUADAS‐2) tool (Whiting 2011). The QUADAS‐2 tool will be applied in four phases: 1) summarize the review question; 2) tailor the tool to the review and produce review‐specific guidance; 3) construct a flow diagram for the primary study; and 4) assess risk of bias and concerns regarding applicability. The tool will provide evaluation across four domains, including patient selection, index test, reference standard, and flow and timing. All domains provide judgement for risk of bias with ratings of 'low', 'high', or 'unclear'. Additionally, the domains of patient selection, index test, and reference standard will evaluate concerns regarding applicability with ratings of 'low', 'high', or 'unclear'. We will classify studies as overall 'low' risk of bias when judged as 'low' across all four domains or 'low' across three domains with one 'unclear' domain. We will utilize a pre‐determined tool (Appendix 2). Any disagreements between review authors will be resolved through discussion with a third review author (GDP).

Statistical analysis and data synthesis

The index test comprises three components: 1) head impulse test; 2) direction‐changing nystagmus; and 3) test of skew. A positive finding in any of the three components will be classified as a 'positive HINTS' examination for overall evaluation. We will generate 2 x 2 tables of TP, TN, FP, and FN and use this data to estimate summary sensitivities and specificities with 95% confidence intervals (95% CI) for the HINTS examination, as well as each of the individual components. We will calculate positive and negative likelihood ratios from the summary sensitivity and specificity data. Data will be visually depicted using forest plots as well as a summary receiver operating characteristic (sROC) curve with 95% confidence region.

We will complete statistical analyses using a bivariate, generalized linear mixed model (Chu 2006). The parameters of the bivariate distribution will be used to create sROC curves (Reitsma 2005). We will calculate the summary estimates via the MIDAS and METANDI modules for StataMP (Stata). We will transcribe the summary estimates to generate Forest plots and sROC plots in Review Manager 5 (Review Manager 2020). For instances where a bivariate mixed‐effects regression model is not feasible due to low study inclusion, then we will describe the study results.

Investigations of heterogeneity

We will plan to investigate heterogeneity for diagnostic tests when data is sufficient (more than 10 included studies). We will investigate heterogeneity between studies with regards to visual examination of forest plots in subgroup analyses. We will investigate heterogeneity by meta‐regression considering specific providers (physician versus non‐physician), time from symptom onset to presentation (less than 24 hours versus greater than 24 hours), reference standard (advanced imaging versus discharge diagnosis), underlying etiology (ischemic stroke versus alternative etiology), and study setting (outpatient versus hospitalized patients). We will also investigate heterogeneity via sensitivity analysis based on study quality and time of completion for the index test. We selected physician versus non‐physician providers given the differences in training. We selected time from symptom onset to presentation to account for the potential in different diagnostic accuracy among early versus delayed presentations. While both reference standards are acceptable, there remains no perfect gold standard, so we will subgroup based upon reference standard. We selected underlying etiology given the potential that an ischemic stroke may differ from a non‐ischemic stroke etiology. We selected study setting given potential differences in populations leading to spectrum bias.

Sensitivity analyses

We will perform a sensitivity analysis of only prospective studies due to the greater potential for confounding among retrospective studies. We will also consider performing a sensitivity analysis based on methodological quality. We will include studies classified as overall 'low' risk of bias from the QUADAS‐2 assessment.

Assessment of reporting bias

We will not conduct an assessment of reporting bias because no standard approach is currently available in diagnostic test accuracy reviews.

Appendices

Appendix 1. MEDLINE search strategy

1. "Sensitivity and Specificity"/
2. Point‐of‐Care Testing/
3. ((point of care or bedside) adj3 (test$ or assessment$ or exam$ or measur$ or procedure$ or diagnos$)).tw,kf.
4. Neurologic Examination/ or Head Impulse Test/
5. ((head adj5 ((impulse or heave or thrust) adj3 (test$ or exam$ or video$ or assessment$ or measur$))) or video‐oculograph$).tw,kf.
6. (HINTS or HITS or vHITS or v‐HITS or h‐HIT or video‐HIT or HITD or (HIT adj3 devic$)).tw,kf.
7. (((oculo‐motor or oculomotor or (eye adj2 movement$) or nystagmus) adj3 (test$ or assessment$ or exam$ or measur$)) or "test of skew").tw,kf.
8. ((eye or vision or visual$ or head$ or saccad$ or ocular or oculomotor or oculo‐motor) adj3 (motion$ or intoleranc$ or movement$ or function$ or direction$ or refixation or fixation or unidirection$ or uni‐direction$ or horizontal or vertical or torsional or bidirection$ or bi‐direction$) adj3 (diagnos$ or test$ or assessment$ or exam$ or measur$)).tw,kf.
9. (((eccentric adj3 gaze) or ((skew or ocular) adj3 (deviation$ or misalign$)) or "vestibulo‐ocular reflex") adj3 (diagnos$ or test$ or assessment$ or exam$ or measur$)).tw,kf.
10. or/1‐9
11. Vestibular Diseases/
12. (acute adj3 vestibular adj3 (syndrome or disease or dysfunction$ or disorder$)).tw,kf.
13. Vertigo/di, et [Diagnosis, Etiology]
14. Dizziness/di, et [Diagnosis, Etiology]
15. exp Nystagmus, Pathologic/di, et [Diagnosis, Etiology]
16. Nausea/di, et [Diagnosis, Etiology]
17. Vomiting/di, et [Diagnosis, Etiology]
18. (dizz$ or vertigo or nausea$ or nauseous or vomit$ or nystagmus or emesis or light‐headed$ or ((walk$ or gait) adj2 unstead$)).tw,kf.
19. or/11‐18
20. 10 and 19

Appendix 2. Quality Assessment Checklist (QUADAS‐2)

Domain 1: Patient selection

Was a consecutive or random sample of patients enrolled?

Yes ‐ Answer 'yes' if stated a consecutive or random sample was used.

No ‐ Answer 'no' if stated an enrollment method was used other than consecutive or random sampling.

Unclear ‐ Answer 'unclear' if there is no mention of the methods for sampling or enrollment.

Was a case–control design avoided?

Yes ‐ Answer 'yes' if case‐control design was avoided.

No ‐ Answer 'no' if case‐control design was not avoided. Study should be excluded from review.

Unclear ‐ Answer 'unclear' if there is insufficient detail to determine if a case‐control design was avoided.

Did the study avoid inappropriate exclusions?

Yes ‐ Answer 'yes' if there are no concerns regarding inappropriate exclusions.

No ‐ Answer 'no' if patients were excluded because they were difficult‐to‐diagnose or included while being easier to diagnose.

Unclear ‐ Answer 'unclear' if there is inappropriate exclusion criteria cannot be determined.

Risk of bias: could the selection of patients have introduced bias?

High ‐ If the answer to any of the signaling questions results in 'no', then rate as 'high'.

Low ‐ If the answer to all of the signaling questions is 'yes', then rate as 'low'. Also, rate as 'low' if only one signaling question is 'unclear' while the remaining are 'yes'.

Unclear ‐ If two or more signaling questions are rated as 'unclear', then rate as 'unclear'.

Applicability: are there concerns that the included patients and setting do not match the review question?

High ‐ If study patients received extensive testing and risk stratification prior to entering the study, then rate as 'high'.

Low ‐ If study patients did not receive extensive testing and risk stratification prior to entering the study, then rate as 'low'.

Unclear ‐ If patient testing prior to enrollment was unclear (e.g. patients were transferred or referred from another center for evaluation of potential stroke), then rate as 'unclear'.

Domain 2: Index test

Were the index test results interpreted without knowledge of the results of the reference standard?

Yes ‐ Answer 'yes' if the results of the index test were blinded or completed by an independent party.

No ‐ Answer 'no' if the results were unblinded or the reference standard results were known prior to completion of the index test.

Unclear ‐ Answer 'unclear' if a determination cannot be made.

If a threshold was used, was it prespecified?

Yes ‐ Answer 'yes' if a prespecified threshold was used.

No ‐ Answer 'no' if a prespecified threshold was not used.

Unclear ‐ Answer 'unclear' if a determination cannot be made.

Risk of bias: could the conduct or interpretation of the index test have introduced bias?

High ‐ If the answer to any of the signaling questions results in 'no', then rate as 'high'.

Low ‐ If the answer to all of the signaling questions is 'yes', then rate as 'low'.

Unclear ‐ If the answer to any of the signaling questions results in 'unclear', then rate as 'unclear'.

Applicability: are there concerns that the index test, its conduct, or its interpretation differ from the review question?

High ‐ If the providers performing the examination were not trained on how to perform the HINTS examination, then rate as 'high'.

Low ‐ If the providers received specific training on how to perform the HINTS examination, then rate as 'low'.

Unclear ‐ If the provider training for performing the HINTS examination is not described, then rate as 'unclear'.

Domain 3: Reference standard

Is the reference standard likely to correctly classify the target condition?

Yes ‐ Answer 'yes' if MRI, PET, or discharge diagnosis was used as the reference standard.

No ‐ Answer 'no' if a reference standard other than the criteria for 'yes' was used. The study should be excluded.

Unclear ‐ Answer 'unclear' if the advanced imaging modality is not appropriately described as the reference standard.

Were the reference standard results interpreted without knowledge of the results of the index test?

Yes ‐ Answer 'yes' if the clinician completing the reference standard was blinded to the index test result.

No ‐ Answer 'no' if the clinician completing the reference standard was aware of the index test result.

Unclear ‐ Answer 'unclear' if it cannot be determined whether the reference standard result was completed without knowledge of the index test. If it is not explicitly stated that advanced imaging or diagnosis was completed without knowledge of the HINTS exam result, then a rating of 'unclear' should be given.

Risk of bias: could the reference standard, its conduct, or its interpretation have introduced bias?

High ‐ If the answer to any of the signaling questions results in 'no', then rate as 'high'.

Low ‐ If the answer to all of the signaling questions is 'yes', then rate as 'low'.

Unclear ‐ If the answer to any of the signaling questions results in 'unclear', then rate as 'unclear'.

Applicability: are there concerns that the target conditions defined by the reference standard do not match the question?

High ‐ If patients did not receive an accepted reference standard (defined as MRI, PET, or discharge diagnosis), then mark as 'high'.

Low ‐ If all patients received the accepted reference standard (defined as MRI, PET, or discharge diagnosis), then mark as 'low'.

Unclear ‐ If the reference standard is incompletely reported, then mark as 'unclear'.

Domain 4: Flow and timing

Was there an appropriate interval between index tests and reference standard?

Yes ‐ Answer 'yes' if the reference standard was completed at the time of the time of the index test. For outpatients, this would be at the index clinic visit. For hospitalized patients, this would be diagnosis for the given admission.

No ‐ Answer 'no' if the reference standard was completed at a time following the index test. For example, the index test was completed during an outpatient visit, but the diagnosis used was from a later visit.

Unclear ‐ Answer 'unclear' if the time frame relation between the index test and the reference standard are not stated.

Did all patients receive a reference standard?

Yes ‐ Answer 'yes' for a reference standard of advanced imaging or discharge diagnosis.

No ‐ Answer 'no' if the reference standard was not applied to all patients.

Unclear ‐ Answer 'unclear' if the study does not mention whether all patients received the reference standard.

Did all patients receive the same reference standard?

Yes ‐ Answer 'yes' if the same reference standard was used for all patients.

No ‐ Answer 'no' if the same reference standard was not used for all patients.

Unclear ‐ Answer 'unclear' if this information cannot be determined.

Were all patients included in the analysis?

Yes ‐ Answer 'yes' if all participants enrolled were included in the accuracy assessment.

No ‐ Answer 'no' if the number of participants enrolled in the study differs from the number included in the accuracy assessment.

Unclear ‐ Answer 'unclear' if there is ambiguity in the number of enrolled and assessed patients.

Risk of bias: could the patient flow have introduced bias?

High ‐ If the answer to any of the signaling questions results in 'no', then rate as 'high'.

Low ‐ If the answer to all of the signaling questions is 'yes', then rate as 'low'. Also, rate as 'low' if only one signaling question is 'unclear' while the remaining are 'yes'.

Unclear ‐ If two or more signaling questions are rated as 'unclear', then rate as 'unclear'.

Contributions of authors

MG: idea synthesis, literature review, protocol design, and protocol drafting.

GDP: idea synthesis, protocol design, and protocol drafting.

JNC: idea synthesis, literature review, protocol design, and protocol drafting.

Sources of support

Internal sources

• Rush University Medical Center, USA

  Salary support (MG, GDP)

• United States Acute Care Solutions, USA

  Salary support (JNC)

External sources

• No sources of support provided

Declarations of interest

MG: none.

GDP: none.

JNC: serves as a consultant for the American Heart Association for team dynamics and resuscitation research.

New