Table 2.
Reproducibility assessment of the study by Gerstung et al.
| Aspect | Item No | Item | Assessment | Note |
|---|---|---|---|---|
| Accessibility (yes/partially/no) | 1a | Data | Partially | Most data available, TCGA clinical data not provided |
| 1b | Is the data (if available) original, processed, anonymized, or simulated? | Anonymized | ||
| 1c | Data dictionary | Partially | Data dictionary incomplete | |
| 2 | Source code | Yes | Documented in the accompanying Supplementary Note | |
| 3 | Documentation of the project | Yes | Supplementary Note and README files provided | |
| 4 | Statistical software | Yes | R (v.3.1.2) | |
| 5 | Software extensions | Yes | Version information listed in the Supplementary Note | |
| 6 | Operating system and hardware layer | Partially | The authors have no access to the LSF environment used by Gerstung et al. | |
| 7 | Can dependencies be set up easily on the reproducer’s computing platform? | No | Dockerfile is not complete enough to allow a rebuild of the original computing environment | |
| 8 | If not, are there any compatibility issues hindering the setup process? | Yes | Today’s R (v.3.1.2) does not support many packages used; conflicts between OS and R (v.3.1.2) occurred | |
| Clarity (yes/partially/no) | 9 | Description of methods | Yes | Documented in the Supplementary Note |
| 10 | Code readability | Partially | Some variable names not self-explanatory or consistent; coding errors spotted; R style guide seemingly not followed; C + + used via Rcpp package | |
| 11 | Inline comments | Partially | Inline comments are helpful however not sufficient; several code lines are commented out but not deleted | |
| 12 | Documentation of custom packages and functions, if applicable | Yes | Two custom packages with documentation provided: mg14 and CoxHD | |
| Code execution | 13 | Is any form of testing on the functions/packages performed? | Partially | R CMD check for CoxHD performed; testing for the analysis code seemingly not performed |
| 14 | (Running analysis code) On mouse-clicks / Minor modifications required / Major modifications with expertise required (e.g. reverse engineering of results) / Impossible to rerun | Major modifications with expertise required | Coding errors, limited cross-platform portability | |
| Implementation of the theoretical methods | 15 | Consistent /Largely consistent / Largely inconsistent / Unable to identify | Consistent | |
| Matching of outputs | 16 | Format of the results | Main paper: figures; Supplementary: tables and figures; Shiny web portal | |
| 17 | Identical with exactly the same results / Same interpretation with deviations in numbers / Inconsistent conclusions / Unable to reproduce the results | Same interpretation with deviations in numbers / Unable to reproduce the results | Minor deviations spotted among the results that could be regenerated, see Supplementary File 2; these did not alter the study conclusions substantially if at all. However, some results remained irreproducible | |
| Overall reproducibility | 18 | Reproducible / Partially reproducible / Irreproducible | Partially reproducible | |
| 19 | Background of researcher(s) performing the assessment | A clinical epidemiologist and a mathematician |