Domain |
Signalling question |
Application in this review |
Domain 1: participant selection |
A. Risk of bias |
1.1 Was a consecutive or random sample of participants enrolled? |
Assess how the individuals screened and/or quarantined as part of the study were determined; where all individuals were screened (e.g. as part of a blanket screening) or where a random sample was selected, a risk of bias is not likely. |
1.2 Was a case‐control design avoided? |
If disease status was used to determine the sample, a risk of bias should be considered. |
1.3 Did the study avoid inappropriate exclusions? |
Any exclusions to screening/quarantine programmes should be justified; however even with justification, exclusions could lead to bias, especially where the screening and disease status of those excluded are unknown. Thus, if no exclusion criteria were applied, the risk of bias is low. |
Comments |
‐ |
1. Could the selection of participants have introduced bias? |
Consider whether bias may have arisen from 1.1 to 1.3 |
B. Concerns regarding applicability |
Describe included participants (prior testing, presentation, intended use of index test and setting) |
Consider those individuals screened, and whether they are representative of individuals likely to be screened during the COVID‐19 pandemic. These studies should be therefore regarded as having a low external validity. |
Is there concern that the included participants do not match the review question? |
See above |
Domain 2: index test(s) |
A. Risk of bias |
2.1 Can we be sure that those identified in index test (true and false positive screening results) were identified by the index test (e.g. automated fever scanner) rather than any other means (e.g. self‐reporting)? |
Consider how those screened positive were determined – all ‘positives’ should stem from the symptom screening and not from any other procedures (e.g. self‐reporting of cases missed by the screening intervention; based on respiratory symptoms). |
2.2 Were the index test results interpreted without knowledge of the results of the reference standard? |
Consider whether, for example, the results of the PCR test were known when symptom or fever screening was applied to individuals. |
2.3 If a threshold was used, was it prespecified? |
Consider for temperature screening, whether the cut‐off for determining acceptable/high temperature was predefined; for symptom screening, consider whether any symptom or a certain threshold of symptoms was used in defining whether an individual was symptomatic and whether this was predefined. |
Comments on risk of bias |
‐ |
2. Could the conduct or interpretation of the index test have introduced bias? |
Consider whether bias may have arisen from 2.1 to 2.3 |
|
B. Concerns regarding applicability |
Describe the index test and how it was conducted and interpreted |
Consider the screening/quarantine programme assessed, and whether it is representative of one likely to be applied as part of screening programmes during the COVID‐19 pandemic. |
Is there concern that the index test, its conduct, or interpretation differ from the review question? |
See above |
|
Domain 3: approach to identify cases and timing |
A. Risk of bias |
3.1 Is the reference standard (the approach to identify and classify 'cases') likely to correctly classify the target condition (is there active infection with SARS‐CoV‐2)? |
Consider whether the approach to identify cases may have missed relevant cases or classified individuals not infected with SARS‐CoV‐2 as a case. Any method other than positive PCR test results can be considered at high risk of bias. |
For the studies using a case‐classification based on a positive PCR test, we assumed the risk of bias due to false positives as low due to the high specificity of the PCR test (in particular if the population is assumed to have a high risk of infection). |
However, there is a considerable risk of false negatives for the PCR test, primarily due to the course of infection (e.g. very low probability of detection in the first days after infection), but also due to inadequate procedures for specimen collection, handling, transportation, or storage (e.g. if only a single test shortly after an infection is applied to a swab sample, the viral load in the individual may not have been high enough for detection, leading to a false‐negative test). |
We therefore assume a high risk of bias in studies, where asymptomatic individuals do not receive at least two PCR tests and symptomatic individuals did not receive at least two PCR tests after symptom onset. |
3.2. Were the reference standard results interpreted without knowledge of the results of the index test? |
Consider whether, for example, the results of the symptom screening were known when the classification was conducted. For PCR tests, where the risk of subjective judgements to have led to a risk of erroneously classifying a test result as negative or positive is regarded as low, this knowledge of the outcome of the index test is still regarded as leading to a low risk of bias. |
Comments on risk of bias |
‐ |
3. Could the reference standard, its conduct, or its interpretation have introduced bias? |
Consider whether bias may have arisen from 3.1 to 3.2 |
Describe the reference standard and how it was conducted and interpreted |
Consider the procedure for determining who receives the reference standard (the PCR test used to identify cases), and whether it is representative of that likely to be applied as part of screening programmes during the COVID‐19 pandemic. |
B. Concerns regarding applicability |
Is there concern that the target condition as defined by the reference standard does not match the review question? |
See above |
Domain 4: flow and timing |
|
4.1. Did all participants receive the reference standard? |
Consider whether all individuals received the reference test (the respective approach to identify and classify ‘cases’; in most cases likely the PCR test). |
For example, if only those who were screened positive (positive index test) and those who developed symptoms during a quarantine observational period were given a PCR test, as this would have led to a high risk of bias due to cases being missed). |
If individuals declined to or for other reasons receive the reference standard (e.g. PCR test), this could lead to cases being missed, which puts the study at a high risk of bias. |
Note: this is independent from 3.1, which evaluates the appropriateness of the approach to classify individuals as cases. |
A. Risk of bias |
4.2. Did all participants receive the same reference standard? |
Consider whether the procedure for identifying cases was the same across all individuals or whether it was applied differently without an adequate justification (e.g. individuals with symptoms receiving a different testing procedure). |
Studies, which used different approaches for classifying cases (e.g. some cases defined based on chest computer tomography and some based on PCR) would be classified as high risk of bias. |
Studies in which the classification of cases is based on multiple PCR tests, we consider a high risk of bias if some symptomatic individuals were treated differently from other symptomatic individuals (e.g. some received more PCR tests than others) and if some of the asymptomatic individuals were treated differently from asymptomatic individuals. |
4.3. Were all participants included in the analysis? |
Consider whether some individuals may have been excluded from the analysis; this would lead to a high risk of bias. |
Is there likely no or a very low risk of attrition bias? |
4.4. Is it possible that the true disease status could have changed between the application of the index test and the reference standard? |
Consider whether individuals may have become infected after the initial screening, e.g. if being quarantined among other infected individuals led to some initially non‐infected individuals becoming infected. If there is a high risk that individuals who were classified as cases were not cases (i.e. not infected with SARS‐CoV‐2) at the time when the index test was applied, this would lead to a high risk of bias. |
Comments on risk of bias |
‐ |
4. Could the participants flow have introduced bias? |
Consider whether bias may have arisen from 4.1 to 4.4 |