| Study ID | Study design | 1. Are the structural assumptions transparent and justified? | 2. Are the structural assumptions reasonable given the overall objective, perspective and scope of the model? | 3. Are the input parameters transparent and justified? | 4. Are the input parameters reasonable? | 5. Has an external validation process been described? | 6. Has the model been shown to be externally valid? | 7. Has an internal validation process been described? | 8. Has the model been shown to be internally valid? | 9. Was there an adequate assessment of the effects of uncertainty? | 10. Was technical documentation, in sufficient detail to allow (potentially) for replication, made available openly or under agreements that protect intellectual property? | Further comments concerning bias and evidence |
| Alvarez 2020 | Compartmental SEIR model with additional states * Model is extended by mild symptoms, presymptomatic transmission, hospitalised cases, ICU cases and deaths * Age‐stratification by context‐dependent contact matrices * Includes contact tracing and symptom‐based isolation * Models Chilean Population |
Yes Model equations are clearly stated and scheme is visualised; one of multiple reports with similar methodology, but sufficiently explained in this report (but references to other reports which may contain further justifications); structure is mostly motivated by intuitive reasoning |
No/minor concerns The model structure as employed is generally sensible |
Moderate concerns Most input parameters are not stated explicitly or explained, but instead with reference to other reports. Not entirely clearly laid out which parameters were used, especially with respect to parameters which have been calibrated; calibration data have been given with source and also visualised |
Major concerns There are concerns with regards to some important parameters employed, as found in their report #3 (e.g. symptomatic contact rate, relative infectiousness between compartments have been assumed). Contact matrices are critical |
Partial Calibrated predictions to case data and death data and similar data sets |
Moderate concerns Calibrated curve fits the data, but only weak dependent validation as there are only two rather simple data sets independent of each other |
No No internal validation |
Moderate concerns No internal validation |
Major concerns There have been no uncertainty analyses reported; only analysis for different scenarios |
Moderate concerns Code has not been reported, but replication might be feasible |
|
| Aspinall 2020 | Bayesian Belief Network (BBN) *Primary schools in England *Focus on number of schools with ≥ 1 infection depending on prevalence |
Partial There is a justification, however not convincing; no argument why BBN is appropriate |
Moderate concerns BBN/hazard model cannot track individuals |
No/minor concerns They are transparent and justified rather well |
No/minor concerns Population parameters are known or distributions including uncertainties were assumed |
No No external validation |
Major concerns No external validation |
Partial Authors refer to a well‐established tool (UNINET) |
No/minor concerns UNINET should be well tested |
No/minor concerns Comprehensive Monte‐Carlo approach, partly expert judgement |
No/minor concerns Comprehensive information, reference to an unpublished programming code file |
|
| Baxter 2020 | Agent‐based modelling study * Outcome at population level in Georgia, USA |
Partial Only reference to previous publications which do not seem relevant |
Moderate concerns Justification in references seems rather convincing, but based on previous models for influenza |
No/minor concerns Only reference to previous publications which do not seem relevant |
Moderate concerns Justification in references seems rather convincing, but based on previous models for influenza, decline because of missing susceptibles seem unrealistic |
No No external validation |
Major concerns No external validation; Decline (it seems to occur because of limited number of susceptibles which is unrealistic. |
Partial No internal validation described. However major parts seem to be based on an established framework. |
Moderate concerns No internal validation |
Major concerns Not reported |
Major concerns No code, description only via references, it is unclear which parts are from with reference. Unclear how many times model was run. Paper written in the style of a quick tech report |
Limited number of susceptibles ≥ unrealistic |
| Bershteyn 2020 | Some kind of simulation model, but not really clear what was done * Some parts may be purely observational results without use of model, which may be applicable |
No Some mathematical model details are scattered around the paper, but the general model structure is mainly unclear |
Major concerns Lack of model structure descriptions justifies major concerns |
Major concerns Input parameters are described every now and then, but their role in the model is mainly unclear |
Major concerns As it is unclear how model parameters are used in the model, there are major concerns to whether they are reasonable. The secondary attack rate seems to be an important parameter, but unclear how it is used. |
No No external validation |
Major concerns No external validation |
No No internal validation |
Major concerns No internal validation. Major concerns due to lack of transparency of approach |
Major concerns There are some uncertainty analyses on the simulation parts, but unclear which uncertainties are covered by these analyses |
Major concerns Replication is impossible given the available descriptions |
|
| Burns A 2020 | Deterministic SEIR‐Modification * Accounts for cohorts (age groups) * Investigates symptom‐based isolation strategies * Time‐dependent infectiousness |
Partial Model is roughly justified with reference to previous studies in the same field. Special properties of this model are justified on base of reasoning. The exact structure of cohorting is mentioned, but never elucidated in detail. Relationships of parameters and states might benefit from more visual representations |
Major concerns State equations seem questionable, for example: "Return to isolation" parameter controls flux out of and into isolation. Although not really mechanistic, model makes a lot of detailed but not well‐founded assumptions which, for example, are based on influenza behaviour; model seems a bit over‐parametrised. A deterministic model can be problematic in the context of smaller systems like schools with rather small age cohorts, since stochastic effects may become important (superspreading and similar occurrences) |
Moderate concerns There is a table of input parameters with some references to sources and if they were calibrated. The transparency of input parameter values is of some concern, as not all are clearly stated in the manuscript (e.g. relative contact rate), some with reference to a repository which has not been checked further. |
Major concerns There are major concerns of the validity of inputs as there are a lot of different parameters needed in the model, but their values and their appearance in the model are not always clear. A 30‐day period of infectiousness for COVID‐19 is at least questionable. As some inputs have been supposedly calibrated from influenza data, the validity of values is compromised. Sources and reporting do not award enough credibility to the many input parameters needed for the model. |
Partial The authors mentioned "validation", but data were only calibrated. |
Major concerns Description of calibration process and the illustration barely sufficient to establish that calibration is successful |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns There is a hint to some kind of parameter uncertainty analysis, but the details are hidden in a repository which was not accessed, should be reported in document due to its importance; results have been presented with uncertainty which arises from uncertain parameters |
Moderate concerns There are links to some repositories with reference to data, but it is not entirely clear whether they contain the study code |
|
| Campbell 2020b | Simple health economic model to calculate the cost of passive and active surveillance testing * Considers Canadian population * Comprises a testing scenario for schools |
Yes Structural assumptions are mechanistic and well explained |
No/minor concerns The study structure is mostly clear and its assumptions are reasonable; partial surveillance scenario with some questionable assumptions (e.g. about test frequency and necessity). Study covers PCR, point‐of‐care tests that are increasingly more relevant |
No/minor concerns Input parameters are all stated with plenty of sources |
No/minor concerns No concerns about validity of input parameters |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
No/minor concerns Most parameters (especially important ones) have been analysed in one‐way sensitivity analyses and visualised in Tornado Plot |
No/minor concerns Model is well described and some code is given in the appendix |
|
| Cohen 2020 | Agent‐based model (COVASIM) for COVID‐19 transmission * Combination with model of school network structure for King County, USA, * Seven school reopening strategies and three different values for infectious cases in the two weeks prior to school reopening are simulated |
Partial Model structure is based on COVASIM which is roughly described. There is not enough information to understand the school network model |
Moderate concerns Majority of model assumptions seem reasonable; school network: only qualitative information provided to understand the assumptions; reference to COVASIM is given, but not enough information is provided concerning COVASIM |
Moderate concerns Parameter values are not stated explicitly but with reference to the methodological paper (COVASIM). Parameter table would have been helpful, some parameters obtained by calibration |
Moderate concerns In general input parameters seem reasonable, but hard to verify with large Agent‐Based Model. R=0.9 is set as an input parameter before school reopening, explanation: schools reopen after slow decrease in infectivity, variation in this parameter would have been good |
No No external validation |
Major concerns No external validation |
Partial COVASIM is an established framework; no internal validation for the student network model |
Moderate concerns Besides the use of COVASIM no internal validation |
Moderate concerns Many assumptions based on COVASIM are not checked by uncertainty analysis; parametre uncertainties: sensitivity analysis for the infectivity of children, susceptibility of children; stochastic uncertainty is presented for the effective reproductive number |
Major concerns Code for COVASIM is available, no code for the school network model, replication seems impossible |
|
| Curtius 2020 | Measurement of the aerosol concentration in two different classrooms: * first classroom without air purifiers * second classroom with air purifiers In order to calculate the risk of onward infection in the two different classrooms and comparison the infection risk model by Lelieveld 2020 is used as a base for the model |
Partial Two parts of the model: 1. model by Lelieveld 2020: model seems reasonable but based on questionable assumptions; 2. measurement of aerosol in the two classrooms: clearly described. For the modelling part, they just take the model of Lelieveld 2020 |
Major concerns Many assumptions based on Lelieveld's model (Lelieveld 2020) but not described in detail; some figures are not comprehensible |
Moderate concerns Input parameters are stated with their respective sources but just one source for almost all parameters, a reduced viral load by the factor ten for children is stated without any source |
Moderate concerns Questionable input parameters, especially parameters concerning the infection risk |
Partial Experimental approach in order to assess their assumptions of the particle concentration levels; no external valdiation for the other part of the model |
Moderate concerns The conducted experiment suggests some external validity for a part of the model |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Uncertainty of measurement devices of purifiers is given, no sensitivity analysis and no parameter uncertainty analysis |
Moderate concerns No code available, with the data available replication of results seems feasible |
It is rather an experimental approach, the modelling part is small and based on references. |
| Di Domenico 2020a | Author description: stochastic discrete age‐structured epidemic model * In its core, the structure is a bit unclear * Models possible Ile‐de‐France school opening scenarios from May to summer holidays |
Partial Although there are many details about the model described, the core of the utilised mathematical model is seemingly never described explicitly, making assessment of quality difficult. There seemingly is another paper from the author in which the same approach is utilised, but also complete descriptions are seemingly missing. Quantitative results and methods from other paper are probably used, but mostly not explicitly stated in this context |
Moderate concerns With the available model descriptions and justifications the model seems to make reasonable and justified assumptions. But as the core model structure is unclear, there is a possible risk of bias as some parts cannot be scrutinised |
No/minor concerns Necessary parameters presumably stated with referenced sources and by a parameter table; some parameters are calibrated. Contact matrices would have been nice to have in the paper. Calibration data are not presented in paper, but presumably in other paper. |
Moderate concerns Parameter values are mostly not a direct cause of concern. Speculation about R value during lockdown phase questionable but probably important. Due to obscured structure, it is unclear if all inputs are stated. |
Partial Model calibration successful for some data, but no true external validation in this paper |
Moderate concerns No true external validation reported |
Partial No internal validation |
Moderate concerns No internal validation |
Moderate concerns Uncertainties and sensitivity analyses of results generally reported. Sensitivity to parameter values was analysed for the relative infectiousness of young children, effectiveness of case isolation and the expected R value during lockdown. Stochastic uncertainties have been considered and visualised. Structural uncertainties presumably not considered and also unclear structure. |
Major concerns Code has not been made available and it might not be possible to replicate results given the descriptions |
|
| España 2020 | Meta‐population model * Based on FRED (Framework for Reconstructing Epidemic Dynamics) * Models population of Indiana * Adjusted for properties of COVID‐19 * Investigates effects of face‐mask adherence and school operating capacity |
Yes Although based on an existing tool, there is a detailed summary of model structure and modifications to account for COVID‐19. Structural assumptions are mostly reasonable as the model is mechanistic. Not fully clear how face masks and school operating capacity are incorporated structurally. |
No/minor concerns Overall, model structure is reasonable. There are some minor concerns due to inexplicit description of incorporation of face mask and school operating capacity effect. Assuming that community level reproduction number does not change is questionable, but appropriate assumption if only school effect should be assessed. |
No/minor concerns COVID‐19 relevant parameters are described in paper and referenced with sources. For other parameters FRED is referenced, but they are mostly not explicitly stated. Data used for calibration is clearly stated and referenced. |
No/minor concerns Stated inputs are mostly reasonable. Authors make use of age‐dependent susceptibility, may be questionable given the extent of justification and its importance. |
Yes Data calibrations are visualised. Results were validated on serological results of cumulative proportions of infected individuals and also stratified for different age groups. |
Moderate concerns Although there are independent assessments of external validity presented, the extent of validation is still rather small with regards to their quality and their agreement. Data calibrations were mostly successful within the presented uncertainties, although there are some concerns. |
Partial Established tool has been used |
Moderate concerns Authors used an established tool, but no specific internal validation |
Moderate concerns Results were presented with credible intervals in all instances and uncertainty has also been visualised. However, due to inherent complexity of the model many structural/parameter uncertainties are not considered which raises concerns about the adequateness of presented credible intervals. |
Moderate concerns Study‐specific code has not been made available. But structure and methods are otherwise described in sufficient detail to possibly replicate results by modifying the base FRED |
|
| Germann 2020 | Agent‐based community simulation of USA * Two levels of working, nine levels of schooling * Some scenarios only for the Chicago region |
Partial Major parts of the model structure are taken from literature, however the description is incomplete |
No/minor concerns There are no obvious problematic assumptions, however assumptions not completely listed |
Moderate concerns Information incomplete, no list of all parameters |
No/minor concerns Information incomplete but no obvious problems |
No No external validation |
Major concerns No external validation |
Partial No internal validation described. However, major parts are based on an established framework |
Moderate concerns No internal validation described. However, major parts are based on an established framework |
Major concerns No uncertainty analyses performed |
Major concerns No code available, description is incomplete |
|
| Gill 2020 | Agent‐based model of schools (children + others) and transport of children |
Yes No concerns |
No/minor concerns No specific concerns |
No/minor concerns Comprehensive justification |
No/minor concerns |
No No external validation |
Major concerns No external validation |
Partial No external validation |
Moderate concerns No internal validation described. However, major parts are based on an established framework. In addition, the simulation results seem more smooth than expected |
Moderate concerns Some sensitivity analyses conducted. They refer to a previous similar study where robustness has been shown |
No/minor concerns No code available, description is comprehensive |
|
| Head 2020 | Meta‐population model for San Francisco Bay area * Especially concerned with effectiveness of school measures * Describes time‐discrete stochastic transmission dynamics * Models relations between pairs of individuals by classifying household/school/grade/class/work/community * Survey to obtain age‐dependent community transmission |
Yes Structural assumptions are well described and mostly justified or at least documented |
No/minor concerns Structure is mostly acceptable; stochastic courses of disease rightfully included; force of infection reasonable; assumptions about interventions are acceptable. Not clear if simulating 1 meta‐individual = 25 real individuals introduces a bias |
Moderate concerns Critical assumption about children susceptibility is well justified by literature. Other parameters are also stated with sources and in table. Important parameter "mean transmission rate" not entirely clear in derivation and value has not been stated. Community contact matrix is not explicitly stated. |
Moderate concerns There are some concerns about the general mean transmission rate and the relative differences between the different transmission classes (work/school/household etc.) as they are critical. Many intervention effectiveness parameters have just been assumed. |
Yes Model has been validated in various instances: * comparison with case data after interventions * comparison with seroprevalence data * household attack rate has been compared to literature * composition of synthetic population has been validated |
Moderate concerns Although external validation is given, the quality and extent of validation is not sufficient to confidently validate model outputs |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Uncertainty in the susceptibility of children and the transmission context during the evaluated scenarios has been assessed. Stochastic uncertainty due to the simulation nature has been assessed by generating 1000 simulation runs. Uncertainties to results are given but they are quite large. Still, due to the many parameters and assumptions in the model there are concerns as to how reliable results are. |
Major concerns Code has not been made available but would likely be necessary to replicate analysis due to its complexity |
|
| Jones 2020 | Poisson regression model * Models total cases in Florida school districts * Covariates: prevalence, percent in‐person enrolment, total district enrolment |
Partial The structural assumptions are stated transparently, but it has not been well justified (although model is simple); almost no references |
No/minor concerns Model seems mostly reasonable, but choice of Poisson regression could have been better justified. Results confirm that predictors all have significant impact |
No/minor concerns Many data sets are mentioned, but which data has been used for regression is not entirely clear. There are references to data repositories. |
Major concerns Besides the minor concerns about the description of employed data, it seems like data for schools with no outbreaks have not been considered. This might introduce major bias. |
Partial By virtue of the model structure, calibration is necessary part of model |
Moderate concerns No rigorous quality of fit measure has been described, but standard errors and significance values for parameters suggest reasonability of structure |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Regression parameters are given with z‐values, two similar data sets have been used. No alternative predictors have been assessed |
Moderate concerns Code has not been made available. Data is supposedly stored in repository and the model is described in sufficient detail to replicate analysis. |
|
| Kaiser 2020 | Network model: simulating the transmission of COVID‐19 in classrooms:
* dividing each class in two cohorts which are taught separately;
* four different cohorting strategies: randomly splitting, splitting by gender, separation optimised by minimising intercohort‐contact out of school, network‐based chains for the out‐of‐school contact as a basis of the separation |
Yes Model structure seems reasonable |
No/minor concerns Out‐of‐school interaction of children is based on a different model and seems realistic; information about the mathematical reasoning for the model is missing |
No/minor concerns Sample: 507 classrooms in England, Germany, the Netherlands and Sweden, data for student interaction by a model of 2010/11 (CILS4EU), this data might be outdated; most of the data with reference to literature; just one source for important parameters Davies 2020 |
Moderate concerns Input parameters seem mostly reasonable, some parameter values are stated through literature, others through theoretical reasoning. Assumed fraction of high‐risk contacts and reduced infectivity of low‐risk contacts. |
No No external validation |
Major concerns No external validation |
No No external validation |
Moderate concerns No internal validation |
No/minor concerns Stochastic uncertainty: 300 simulations for each classroom were performed and the average result is given, no further evaluation of stochastic uncertainty; parameter uncertainties are checked for transmission, out‐of‐school interaction and proportion of infections by using different plausible values; uncertainties for parameters concerning the infection are not assessed; structural uncertainties are not assessed but network plausible |
Moderate concerns No code available, description rather comprehensive, replication of model might be difficult |
|
| Keeling 2020 | Complex SEIR‐based ODE model for UK with: *fine‐grained age stratification *school/work/household transmission *undetected/detected cases *compliance‐dependent effect on contact matrices |
Partial Larger ODE model makes it difficult to examine the complete dynamics, visualisation would have been helpful. It is not always clear how analyses exactly have been conducted. There are references to a previous paper with more detailed methodology, but also not perfectly detailed. |
No/minor concerns No direct concerns about specific points. Generally, an overwhelming amount of implicit assumptions to consider due to complexity of model and some lack of descriptions |
Moderate concerns Sources of data and parameters seem to be mostly stated. Parameter table is given, mixing matrices and age‐dependent parameters as figures. Many parameters calibrated from data, but calibration data are not shown and not entirely clear. |
No/minor concerns There are some concerns since it is not clear which data fitting calibrated the parameters (there are some descriptions, but lack of reporting). |
Partial There is dependent validation due to model calibration, but there is limited information about how well model is calibrated to data. The model calibration is done in another paper. |
Moderate concerns Calibration in referenced paper by same author |
Partial There is some validation by authors reported at the end of paper, but no processes reported |
Moderate concerns No internal validation conducted, but model is complex so it would be necessary to check |
Moderate concerns Uncertainties have been partially reported from parameter posterior distributions, covering stochastic and parameter uncertainties. However, uncertainty for some parameters seem rather small. There are some instances in which possibly important values are assumed to be fixed (age‐dependent mixing matrix, effect of lockdown on mixing matrices). Due to its specific model structure, study would have benefited from an analysis by use of a different model structure |
Major concerns Code has not been made available and the way data that are presented will presumably complicate replication attempts |
|
| Kraay 2020 | SIR‐based modelling study *Focus on transmissions via hands and fomite (surface) touching |
Partial Stated "previously described" but no reference provided |
Moderate concerns Only deterministic, very simplified structure |
No/minor concerns Mainly justified by influenza and rhinovirus values |
Moderate concerns Partly taken from influenza/rhinovirus |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
Major concerns Sensitivity analysis for only a few parameters |
Moderate concerns No code available, description rather comprehensive |
|
| Landeros 2020 | SEIR‐based ODE model for the USA * Three different school opening scenarios: reopening at full capacity, allowing half of the students to attend school, rotating cohort (students are divided into 3 cohorts and 2 of them are allowed to attend school at the same time) * Effect on the reproduction number (R) and prevalence is simulated under these three possibilities and compared to the impact of people > 18 years on R and cumulative prevalence of COVID‐19 |
Yes Model structure is clearly stated and justified; equations are based on mathematical reasoning |
Major concerns Model assumptions are simplistic; cohorting strategies for children because of school reopening strategies, but it is unreasonable to have different cohorts in the model for adults as well; model is stated to apply to school communities rather than states |
No/minor concerns Input parameters are justified, literature is given for most of them; child‐to‐child contact rate at school is given without any source |
Major concerns Latent, infectious and incubation period are justified by literature. Weak justification for other parameters such as same values for children and adults for transmission and their latent and infectious period and no source for the multiplier for increased child‐to‐child‐contact c = 10. Input parameters for the transmission rate are highly unspecific, they have a wide range. |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Parameter uncertainty for transmission rate is assessed by large range of different values for said rate. Structural uncertainties are not discussed, although probably important |
No/minor concerns Code available from the author by request; description is comprehensive |
Wide range for the input parameters ≥ no significant result |
| Lazebnik 2020 | Hybrid model: SIRD type temporal dynamics and spatial dynamics for home, school, workplace * Additional compartments: age ‐ children (< 13 years) and adults |
Partial There is a good overview of other studies and their results, motivating the approach. ODE part is described extensively and transparently. Spatial part seems to be a stochastic simulation, but description lacks depth to understand the mechanics involved. |
Moderate concerns Generally, the model adopts features which possibly could produce sensible results due to age stratification and differences in mixing patterns due to different physical locations. But according to the model, children above 13 years would have the properties of adults, i.e. go to work, 2 class age stratification might not be enough. Model is just a forward simulation of input parameters, which requires great care concerning the inputs and their applicability as well as a reliable model structure. Regarding this aspect, there are concerns about the validity of the model. Spatial part can not really be fully assessed with the available information. |
No/minor concerns Input parameters are stated with their respective sources in most cases. The number of meeting events is set to one per hour, without further commentary. |
Major concerns There are significant concerns about the model inputs due to their significance in generating the model results. The inputs are mainly parameters from other studies, such that their reliability in this study are not guaranteed as they are not calibrated against data. Some parameters seem odd: why would children not be able to infect other adults, but other children? (beta_ac,beta_cc) This should presumably be property of the spatial structure, not of the transmission parameter. The derivation of beta_ca as reported is questionable, since beta incorporates infection as well as contact probability, but the derivation only covers infection probability reliably |
Yes Daily R0 from data was compared with R0 from model for a two‐week span before and after school closure. |
Moderate concerns It was shown that the model can in some way approximately reproduce the case numbers in a small time frame. It is not reported to which extent this is really an independent validation. Although better than simple calibration, this is still a weak validation. There have been some comparisons to other modellers' results. |
Partial There are some sanity‐check type analysis from a mathematical standpoint concerning the equations, but from a computational standpoint it is unclear whether the implementation is right |
Moderate concerns Not convincingly validated |
Major concerns Uncertainty has mostly not been assessed, even if it would have been important due to nature of the forward simulation type model. Stochastic Uncertainty was partially assessed as some R2values for result fits have been specified. Parameter uncertainty has not been assessed. Structural uncertainties were not considered, although there has been a discussion of other model structures. |
Major concerns Code has not been made available. Description of spatial stochastic model part lacks in‐depth explanation such that it might not be possible to reproduce model |
|
| Lee 2020 | Simple age‐stratified estimation for basic reproduction number (R0) based on assumed SIR model * Considering different frequencies of contacts among age groups * Impact of different susceptibilities among age groups is assessed |
Yes Model clearly described. |
Moderate concerns Within the limits of SEIR model |
Moderate concerns Sparse details. |
Moderate concerns Sparse details. |
No No description of external validation. |
Major concerns No description and based on hypothetical situation, not a particular context. |
No Not described |
Moderate concerns Not described. |
Moderate concerns Tested 5 different scenarios of children's % susceptibility from 35 to 60% |
No/minor concerns Model available on Github. |
Simple model, but large influence of the contact matrix on the outcome. Contact matrix just roughly described |
| Lyng 2020 | SIR model analysing different test/surveillance strategies * Linked to two observed prevalences in population * No stochasticity, no agents, basic reproduction number (R0) = 2.5, institution = subset of 1500 people |
Yes Information in paper and supplement seem to be complete |
Major concerns Deterministic with fixed R0, very simplified model structure, scope: one initial condition (1.35 infections) and two prevalence scenarios |
No/minor concerns Justification sufficient, however only very few parameters required |
Major concerns Decrease due to limited number of susceptibles, R0=2.5 |
No No external validation |
Major concerns No external validation |
Partial No internal validation described, but code (partly) and online simulator available for testing validity |
Moderate concerns No internal validation described, but code (partly) and online simulator available for testing validity |
Major concerns The weakest part of the study is missing analysis of uncertainty. Predicting costs and effectiveness at an absolute level without uncertainty or sensitivity analysis poses a serious risk. |
No/minor concerns Code is partly available, online simulator available |
Limited number of susceptibles ≥ unrealistic |
| Mauras 2020 | Agent‐based SEIR with contact networks: * investigates probabilities of outbreaks after one index case |
Yes Good and convincing |
No/minor concerns Comprehensive justification, realistic structure |
No/minor concerns Justification sufficient |
No/minor concerns |
Partial Comparison with some specific findings in other studies |
No/minor concerns External validation as good as possible done by comparing with literature |
Partial No explicit internal validation procedure but a very comprehensive set of analyses were done that indicate validity |
No/minor concerns No explicit internal validation procedure but a very comprehensive set of analyses were done that indicate validity |
No/minor concerns Sufficient analyses by evaluating parameter sensitivity and dependency on model assumptions |
No/minor concerns Code available on github, results seem reproducible |
The model focus is on temporal evolution of single index cases within school/workplace. They consider the probability of getting an outbreak (≥ 5 secondary cases). The effect to the population is not the primary scope of the model. |
| Monod 2020 | Bayesian model for transmission dynamics in the USA * Age‐stratified contact‐and‐infection model, * Impact of different age groups to infection dynamics is estimated * Interaction for different age groups is based on mobile phone data, then SARS‐CoV‐2 transmission, infections and deaths are estimated |
Yes Relative mobility levels for the different age groups: mobility between February and August compared to a baseline; mobility is attributed to mortality data to fit the model; mathematical approach is clearly described |
No/minor concerns Model assumptions are justified; limitations: population structure except age is not completely accounted for, young children without phone cannot be followed up, but source for their mobility input data is given; mobility of population depends on a lot of external factors |
No/minor concerns Reference for input parameters is given; two sources for network data are given |
No/minor concerns Input parameters seem reasonable for the US, but strongly depending on the population structure |
Yes Validation for the interaction of individuals by data of a second mobile phone provider; predictions of the model are compared to reported cases of COVID‐19; calibration for the cumulative number of deaths seems reasonable |
Moderate concerns Age‐stratified death data closely matches the model predictions; number of reported COVID‐19 cases compared to the prediction of the model increases, but explanation is given (increased testing); calibration as kind of dependent validation |
No No internal validation |
Moderate concerns No internal validation |
No/minor concerns Credible intervals for key outcomes are given (e.g. R0, onward spread, contribution to infection transmission); parameter uncertainties: sensitivity analysis for the age‐stratified infection fatality ratio; one reference to a similar model, besides that no assessment of structural uncertainties |
No/minor concerns Code available on Github, MIT license is needed |
|
| Munday 2020 | Network model describing transmission between schools * Transmission probability model showing the interaction of schools and households in England * Outbreak probability for six different school reopening scenarios is modelled |
Yes Majority of model assumptions are stated through equations, visualisations might have been helpful |
Major concerns Model assumptions seem idealistic, because the network is simplistic: it accounts for household and schools, other population structures are neglected. Spread between schools is seemingly mediated by infection between siblings in households which seems questionable |
No/minor concerns Source of information for the network of schools in England is given. Parameters are complete, but only a small amount of input parameters are used. |
No/minor concerns Input parameters are reasonable |
No No external validation, but reference to other studies who came to similar qualitative results |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Parameter uncertainty: sensitivity analysis for the reproductive number (R) and for the within‐household transmission probability; stochastic uncertainty: credible intervals are given, 100 simulations in order to account for stochastic uncertainty; no structural uncertainty analysis, although this is needed to justify the structure |
Major concerns No code available, with the data available replication of results might be difficult |
|
| Naimark 2020 | Agent‐based SEIR‐based simulation model * Model to calculate cumulative COVID‐19 cases for six different scenarios: schools remaining closed and schools being reopened in combination with three different non‐pharmacological intervention (NPI) measures; * Hypothetical population of one million individuals based on the characteristics of the population of Ontario, Canada, calibrated for the first and second COVID‐19 wave |
Yes Model structure is stated with reference to the supplementary material; clear visualisation in the supplementary material; reference to a similar model in another study |
No/minor concerns In general it seems reasonable to combine school reopening and schools remaining closed with different NPI measurements; infectiousness of children might be different to adult's infectiousness |
No/minor concerns Input parameters are transparent and justified, table for key parameters with sources is given |
No/minor concerns Input parameters seem to be reasonable, parameters are calibrated or with reference to literature |
Partial Calibration and recalibration for the first and second wave of COVID‐19 (dependent validation) |
Moderate concerns Besides the data used for calibration, no proof that the model fits to external data as well |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Stochastic uncertainties are checked by several simulations, credible intervals are given for stochastic uncertainties; parameter uncertainties are checked by the different scenarios, besides that they are not checked |
No/minor concerns No code available, description rather comprehensive |
|
| Panovska‐Griffiths 2020a | Agent‐based SEIR‐model (COVASIM) * Analysed impact of two different school opening scenarios and three ways of testing on reproduction number (R), incidence and death of COVID‐19 * Second simulation with 50% infectiousness of children compared to older ages * Two possible strategies for reopening schools: full‐ and part‐time with 50% attendance, combined with three types of testing |
Yes Model structure is clearly stated and justified, used COVASIM as a basis of model (briefly described) |
Moderate concerns It is reasonable that reopening of schools is proportional to return to workplaces, effect of decisions of policy makers on this topic is neglected; 14‐days complete isolation of people tested positive might be idealistic; prediction until end of 2021 questionable |
No/minor concerns Input data are stated and source is publicly available for confirmed cases and deaths, referring to COVA for other model parameters; updates of COVASIM are integrated into the model |
Moderate concerns In general the input parameters are reasonable; it is referred to the UK Government's COVID19 dashboard; calibration of some parameters; some concerns because model has a lot of parameter inputs |
Partial Dependent validation for the confirmed cases and deaths, with data of UK Government's COVID‐19 dashboard; but these data were also used to build the model, no other external validation |
Moderate concerns Apart from the dependent validation no external validation described |
Partial COVASIM is an established framework |
Moderate concerns COVASIM is an established framework, no other internal validation |
Moderate concerns Assessment for the effects of uncertainties for deaths, R and incidence of COVID‐19; several simulations in order to account for stochastic errors, shown by 10% and 90% quantiles (but only 10 simulations); different scenarios for test‐tracing and school reopening seem reasonable; parameter uncertainties: two different parameters for children's infectiousness, besides that parameter uncertainties are not assessed; structural uncertainties are not further assessed |
No/minor concerns With the given data, replication of results seems possible, Code for COVASIM is available |
|
| Panovska‐Griffiths 2020b | Agent‐based model based on COVASIM, evaluating the impact of face coverings in the UK, number of new infections for different scenarios: * no mask wearing at schools but community mask wearing * mask wearing at secondary schools and community mask wearing Considered two different levels of effective mask coverage |
Yes Model structure seems reasonable, extensions to COVASIM sufficiently described; not enough information about COVASIM |
Moderate concerns It might not be reasonable to predict a pandemic until 12/2021, only one mask‐wearing scenario at school is modelled and compared to no mask‐wearing at school |
No/minor concerns Illustrative table for the input parameters, COVASIM‐based parameters and calibrated parameters are stated |
Moderate concerns Some concerns because of the many input parameters of COVASIM |
Partial There is no external validation but model calibration for the COVID‐19 cases with case data and death data for the UK |
Moderate concerns Data have been calibrated; calibration c |
Partial COVASIM is an established framework |
Moderate concerns COVASIM is an established framework, no other internal validation |
Moderate concerns Stochastic uncertainties: several simulations are done and 10%/90% quantiles are given, stochastic uncertainty is extremely large; uncertainty of input parameters: different values for effectiveness of mask wearing; no assessment of structural uncertainty |
No/minor concerns Code for COVASIM is available, code for the rest of the model is available on github |
|
| Phillips 2020 | Agent‐based simulation of one school/childcare facility embedded in the community * Basic simulation approach, homogeneous mixing based on household/class/school * Investigates allocation of children and educators to classes |
Yes The model structure is documented and justified in most instances. Unclear whether transmission probability is understood correctly, beta as well as contact matrices have been described as the probability of transmission. |
Major concerns Model assumptions might be too simplistic as small scale of model highlights importance of network effects. Homogeneous mixing is argued by aerosol transmission, however this would contradict the assumption of strongly age‐dependent transmission probabilities. As understood by reviewer: transmission probability approximately proportional to class size, might not be expected as contacts of children might not increase proportionally with larger class size. Immediate detection of symptomatic individuals and perfect compliance with no household transmission in isolation is questionable (only 5 classrooms and 1 school) |
No/minor concerns Input parameters have been stated with sources and some were additionally clarified with explanations. For community transmission an under‐ascertainment factor of 8.45 has been assumed without justification. Although hinted at in the text, different infectiousness of children compared to adults has seemingly not been analysed. |
Moderate concerns Transmission probabilities were calibrated to produce a household attack rate of 15% based on only one study, for the class/school the transmission rate has been scaled down somewhat arbitrarily or at least not convincing |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns There were several sensitivity analyses on important parameters. Uncertainties have been generally visualised, in some instances it is not clear whether standard error of the mean or standard deviation of results is given. Error bands which lead to negative proportions of infected individuals indicate flawed uncertainty analysis. Uncertainties generally large, indicates that choice of outcome variables is not perfect (fractions between strategies more relevant than absolute values) |
Moderate concerns Code not available, but data and method might be sufficiently described to allow for replication |
|
| Rozhnova 2020 | Model for the Netherlands, effect of opening/closing schools on effective reproduction number (Re), informative epidemic data (random cross‐ section, not reported cases with symptoms) |
Yes Justification is comprehensive |
No/minor concerns The assumptions are reasonable |
No/minor concerns Justification is sufficient |
No/minor concerns Estimation of parameters using Bayesian approach (priors seems reasonable), reliable methodology, negative binomial observations assumed |
No No external validation, some literature mentioned |
Moderate concerns No independent external validation, but real and very informative data used for parameter fitting, agreement of model and data shown |
No No internal validation |
Moderate concerns No internal validation, but the methodology was applied previously |
No/minor concerns Reliable methodology for uncertainty analyses applied |
No/minor concerns Code available on github, reproducibility seems given |
|
| Shelley 2020 | Deterministic SEIR model stratified into town and different cohorts within a school * Adds preclinical and subclinical infectious states |
Partial Model structure is mostly clear, some lack of justifications. Exact implementation of testing and quarantine in the model not totally clear and neglected in results/discussion |
Major concerns It is doubtful if this deterministic model of such a non‐closed system starting from one seed infection can properly describe infection dynamics; mass testing fraction is randomly drawn between 0 and 1; high sensitivity of results to the first seeded infection implies practical lack of robustness of deterministic approach; beta has seemingly not been adjusted for the change of magnitude introduced by transmission matrices |
Moderate concerns Epidemiological parameters have been set to Centers for Disease Control (CDC). Effect of cohorting has been chosen without quantitative justification |
Major concerns It is conceivable that form of transmission matrices which have not been sufficiently justified have a major impact on results. Role of mass testing which is chosen to random degrees is unclear |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation |
Moderate concerns Parameter uncertainty has been investigated probabilistically. Transmission matrices have not been subject to uncertainty analyses. There are concerns that the simple model structure can not describe the real dynamics, so an analysis of alternative model structure would have been adequate. |
Moderate concerns Code has not been made available but model is comparably simple. Given information might enable replication of model, but unclear implementation of testing and quarantine. |
|
| Sruthi 2020 | Machine‐learning algorithm to disentangle effects of different non‐pharmacological interventions (NPIs) in Switzerland cantons |
Partial Much of the structure is hidden away in an AI‐type algorithm |
Major concerns As far as it can be addressed the assumed structure seems reasonable. Many of the assumptions are impossible to assess given the information in the study. |
No/minor concerns Algorithm parameters are specified; not many more parameters as it seems. |
No/minor concerns Since model inputs are fairly straightforward, there are barely any problems. A minor concern would be the input of recovery time which scales the reproduction rate. |
Yes Five‐fold cross validation |
Moderate concerns Cross‐validity seems to suggest that weekly infection rates can be predicted well if case numbers are high enough. No other forms of validation reported. |
Partial No internal validation, but cross‐validation |
Moderate concerns Functionality of cross‐validation suggests that model is functional in some sense |
Moderate concerns Uncertainties were reported, but they likely do not span varying structural assumptions which may have significant impact on the reproduction rate contributions. |
No/minor concerns Code and source data available |
|
| Tupper 2020 | Agent‐based/stochastic SEIR model of in‐class transmissions: * focus on large clusters (super‐spreading events) |
Partial Weakly justified, but based on a rather widely used model structure |
Moderate concerns Only children, only within classroom considered |
No/minor concerns Mostly justified by literature |
Moderate concerns No obvious issues, but weak justifications for many parameters |
No No external validation |
Major concerns No external validation |
No No internal validation |
Moderate concerns No internal validation done, results look plausible |
Major concerns Only sensitivity analysis for few parameters. These show large impact on results. |
Moderate concerns No code available, description rather comprehensive |
|
| Williams 2020 | COVID meta‐population model for Seattle * Based on CORVID which is based on FluTe which simulated influenza * Analysis of different test and isolation strategies |
Partial Justifications are shifted to the method papers, but mostly understandable there. Descriptions could have been more technical and detailed. Unclear how tests/symptomatic cases averted was calculated |
Moderate concerns There are some concerns as structure is ultimately based on influenza model, with some natural history of disease modifications introduced for COVID‐19. Because model is meta‐population model, it is difficult to verify that structure is reasonable, mechanistic to a high degree. Simulation of results until end of epidemic is rather unreasonable for assessing outcomes, as this creates a large degree of uncertainty. |
Major concerns It is difficult to gather all model inputs, as most of it is not contained in this paper. Additionally, it is difficult to see how much of up‐to‐date parameter knowledge was used in the simulations |
Moderate concerns There are no obvious flaws, but given the paper information this is impossible to assess without looking into code files |
No No external validation |
Major concerns No external validation |
Partial Model is based on existing published framework |
Major concerns Model is based on existing published framework. But the given outputs are not explicitly validated. The almost equal infection peaks for different simulations are atypical for agent‐based models. |
Major concerns Minimal assessments were provided, some instances of different seeds and different R0 analysed. But model still contains a great deal of uncertainties with respect to structural assumptions and implicit model parameters which are hidden. |
No/minor concerns Code and data are available in repository |
|
| Zhang 2020 | Modelling of four Chinese cities; SIR model and with contact matrices based on diaries/questionnaires via phone; analysis only based on reported contacts; most of the information is from reported contacts not from modelling; only "schools open without any containment measures" versus "schools closed" considered |
Yes Justification is sufficient |
Moderate concerns Self‐reported contacts of study participants play a major role in the model |
No/minor concerns Contact matrices are justified, SIR model parameters only partly justified (it seems to be used only for calculation of R0 not for simulating the epidemics) |
Major concerns Self‐reported contact matrices might be strongly biased, estimation of some parameters of SIR model is not described |
Partial Comparison with mobility |
Major concerns No external validation for the important results, i.e. prediction of R0 or reported infections |
No No internal validation |
Moderate concerns No internal validation, but comprehensive analyses that partly indicate reliability, no comparison of SIR model with data about infections |
Moderate concerns Uncertainty of count matrices is reliable, uncertainty from SIR model not considered |
Moderate concerns No code available, role of SIR model not entirely clear, other parts are sufficiently described |
Transfer of results from China to Western countries unclear. Most information is from reported contacts. These reported contacts (via phone calls) might be unreliable. |
Official websites use .gov
A
.gov website belongs to an official
government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you've safely
connected to the .gov website. Share sensitive
information only on official, secure websites.