Table 1.
Critical Appraisal of Methodological Quality and Quality of reporting (CRIME-Q): categories, descriptions, and potential bias/impact on studies
| Category | Type | Questions and clarification | Potential for bias | |
|---|---|---|---|---|
| 1X | Peer review | QoR | Did the paper undergo peer review prior to publication? Peer review might be useful for detecting errors or fraud. Yes/No |
With peer review: Bias against negative studies Without: potential for errors and/or fraud |
| 2X | Bench-top/laboratory work related to establishing model—reporting | QoR | Was the study’s bench-top protocol sufficiently described (transparent, reproducible)? If e.g., cells are involved, did the study e.g., present incubator settings, description in detail of how the cells were treated (transfection, irradiation, etc.)? or describe how the cells were handled? Or for instance, did the study describe how to obtain a certain genetic model given there is no commercially available animal model? Yes/Partly/No | In vivo results are highly influenced by in vitro/bench-top part of the study. If not transparent and reproducible this lessens the usability of the study |
| 2Y | Bench-top/laboratory work related to establishing model—methodology | MQ | Was the bench-top protocol feasible and well performed in relation to the experiment? Was it likely that the intended aim could be obtained based on the bench-top method? Yes/Partly/No. Studies with poor reporting (2X) will have difficulty gaining a high 2Y because of low transparency and the ability to assess method quality | If the bench-top protocol was not feasible, TTRs may be misleading, and readers should use papers with caution |
| 3X | Animals—reporting | QoR | Were the animals used in the experiment sufficiently described? Were all parameters: Type, breed, age, weight, and manufacturer sufficiently described? If type, age, and weight were sufficiently described OR if weight was missing, but the manufacturer was included a Yes was given. If only partly described, then Partly. and if we were not able to correctly identify the animals No was given. Yes/Partly/No | The importance of the description of animal type cannot be understated since the immunological profile differs from strain to strain, which influences results |
| 3Y | Animals—Methodology | MQ | Did the study use similar baseline characteristics for the animals (age, weight, type)? Yes/Partly/No. Studies with poor reporting (3X) will have difficulty gaining a high 3Y because of low transparency and the ability to assess method quality | If animals were not homogenous, study results might vary, for instance, low weight might result in poorer survival, which skews results |
| 3Z | Selection bias (baseline characteristics) (SYRCLE Item 2) | RoB | Was the distribution of relevant baseline characteristics balanced between groups? I.e., was the distribution of e.g., male:female ratio, species, strain, age, and weight equally distributed throughout groups? Yes/No/Unclear/NA. Not applicable to studies using only one group | Unequal groups in intervention studies can skew results – introduces variables that potentially affect study results |
| 4Y | Sample size | MQ | Did the study include a calculation of sample size? Describe how it was calculated – at what power? Was it appropriate and well performed? Yes/Partly/No | Studies may prove under-/overpowered in terms of drug efficacy if too few/many animals were used |
| 5X | In vivo design and performance—reporting | QoR | Description of the in vivo study part: Were the surgery, implantation/injection method, and duration (whole experiment) sufficiently described? Is the study transparent and reproducible? Yes/Partly/No | The results will be difficult to replicate if the study is poorly described. Meaning the study is difficult to properly be assessed as a useful base for further research |
| 5Y | In vivo design and performance—methodology | MQ |
Did the method seem feasible and well performed concerning the study’s aim and outcome and in contrast to other known literature? Is it likely that the in vivo study design influences the results—incomprehensive/insensible method? Yes/Partly/No Studies with poor reporting (5X) will have difficulty gaining a high 5Y because of low transparency and the ability to assess method quality |
A poor methodology can skew results making conclusions in relation to aims obsolete |
| 5Z(1) | Selection bias (Sequence generation) (SYRCLE Item 1) | RoB | Was there a description of allocation (the process by which experimental units are assigned to experimental groups)—And was it appropriate? Not applicable to non-intervention studies. Yes/No/Unclear/NA | Unequal groups in intervention studies can skew results—introduces variables that potentially affect study results |
| 5Z(2) | Performance bias (Random housing) (SYRCLE Item 4) | RoB | Were the animals randomly housed during the experiment? Yes/no/unclear. Not applicable to non-intervention studies. Yes/No/Unclear/NA | Some types of experiments are influenced by the location of housing, hence random assignment of placement could negate these issues |
| 5Z(3) | Detection bias (Random outcome assessment) (SYRCLE Item 6) | RoB | Were animals randomly selected for outcome? For instance, If human endpoints (i.e., poor conditions, weight, etc.) were met and the investigators were not blinded, then the outcome cannot be assessed randomly. Not applicable to non-intervention studies. Yes/No/Unclear/NA | Bias toward assessing intervention effect size |
| 6X | Compliance with animal welfare regulations | QoR | Did the study comply with any animal welfare regulations? Yes/Partly/No | Assurance of proper animal care throughout the study. Also important in terms of survival studies (human endpoints vs. death) |
| 7X | Blinding | QoR |
Was the study blinded in any way? Was the outcome assessed in a blinded fashion? Were the animals randomly selected across all groups of e.g., intervention? Were the investigator or animal handlers blinded? More specific blinding is listed below in 7Z(1–3) Yes/Partly/No |
Unblinded administrator of intervention can skew results in both a positive and negative direction |
| 7Z(1) | Performance bias (Blinding) (SYRCLE Item 5) | RoB | Describe all used means, if any, to blind trial caregiver and researchers from knowing which intervention each animal received. Yes/No/Unclear/NA. Not applicable for non-intervention studies, however, it could be applicable for instance in xenograft studies, where multiple patient samples were used | Animal handling may be affected by unblinded study design |
| 7Z(2) | Allocation bias (allocation concealment) (SYRCLE Item 3) | RoB | Could the investigator allocating the animals to intervention or control group not foresee assignment? Yes/No/Unclear/NA. Not applicable to non-intervention studies. Yes/no/unclear/NA. This could be applicable for instance in xenograft studies, where multiple patient samples were used | In relation to 7Z (1). Selection, handling, and treatment of animals may be affected if allocation concealment was not adequately performed |
| 7Z(3) | Detection bias (blinding) (SYRCLE Item 7) | RoB | Was the outcome assessor blinded? and could the blinding have been broken? Describe all measures used, if any, to blind outcome assessors from knowing which intervention each animal received. Were the outcome assessment methods the same in each group? Yes/No/Unclear/NA. This could be applicable to instance in xenograft studies, where multiple patient samples where used | Measurement of the outcome can be over-/underestimated if proper blinded outcome assessment was not performed |
| 8X | Congruency between methods and results | QoR | Did the study present all their findings based on the methods described? Is there congruency between the method and results sections? Yes/Partly/No | Presenting results in which methods are not described is not transparent and replicable and should be interpreted with caution |
| 8Z(1) | Attrition bias (incomplete outcome data) (SYRCLE Item 8) | RoB | Describe the completeness of outcome data including attrition and exclusions from the analysis and were incomplete outcome data adequately described? Were all animals included in the analysis and if not, was it described why they were not included? Yes/No/Unclear | Attritions and/or exclusions should be clearly described, i.e., the number of animals used. If not, study results become difficult to assess. Poor replicability and transparency |
| 8Z(2) | Reporting bias (Selective outcome reporting) (SYRCLE Item 9) | RoB | Was the study protocol available (require a description of protocol location in the record) and were all of the study’s pre-specified primary and secondary outcomes reported in the manuscript? Was the study protocol not available but was it clear that the published report included all expected outcomes (i.e., comparing methods and results sections)?The study report fails to include results for a key outcome that would be expected to have been reported for such a study, i.e. tumor-take rate in transplantation experiments. Yes/No/Unclear | Congruency between results and methods should be carefully described to avoid reporting bias. If key outcomes for a certain method were not described, study validity, transparency, and replicability become difficult |
| 9X | Presentation of limitations | QoR | Did the study contain a section of limitations, or did they comment on the limitations of the study in relationship to in vitro and/or in vivo subparts? Yes/Partly/No | No study is without limitations, and it is paramount to present them to the reader for transparency’s sake |
| 10X | Statement of potential conflict of interest | QoR | Did the study contain a statement of potential conflicts of interest? Yes/No | Potential conflicts of interest can skew results i.e. if an investigator has a method patent or is paid by a certain pharmaceutical company, hence it is important for transparency’s sake to include it in the study |
| 10Z | Publication bias (influence) (SYRCLE Item 10) | RoB | Inappropriate influence of funders or biased by companies. Was the study free of inappropriate influence from funders or companies supplying drugs or equipment? Did the authors declare a direct conflict of interest in relation to the study? Yes: Conflict of interest statement with no conflicts of interest. Yes/No/Unclear | Publication bias – Negative results will be less likely to be published if inappropriate influence of funder or biased companies occur |
QoR: Quality of Reporting; MQ: Methodological Quality; RoB: Risk of Bias; TTR: Tumor take rate; NA: Not applicable