Table 1.
Key steps and decisions in the planning, coding, analysis and reporting of simulation studies
| Section | |
|---|---|
| planning | 3 |
| Aims | 3.1 |
| · Identify specific aims of simulation study. | |
| Data‐generating mechanisms | 3.2 |
| · In relation to the aims, decide whether to use resampling or simulation from some parametric model. | |
| · For simulation from a parametric model, decide how simple or | |
| complex the model should be and whether it should be based on real data. | |
| · Determine what factors to vary and the levels of factors to use. | |
| · Decide whether factors should be varied fully factorially, partly factorially or one‐at‐a‐time. | |
| Estimand/target of analysis | 3.3 |
| · Define estimands and/or other targets of the simulation study. | |
| Methods | 3.4 |
| · Identify methods to be evaluated and consider whether they are appropriate for estimand/target identified. | |
| For method comparison studies, make a careful review of the literature to ensure inclusion of relevant methods. | |
| Performance measures | 3.5, 5.2 |
| · List all performance measures to be estimated, justifying their relevance to estimands or other targets. | |
| · For less‐used performance measures, give explicit formulae for the avoidance of ambiguity. | 5.2 |
| · Choose a value of n sim that achieves acceptable Monte Carlo SE for key performance measures. | 5.2, 5.3 |
| coding and execution | 4 |
| · Separate scripts used to analyze simulated datasets from scripts to analyze estimates datasets. | |
| · Start small and build up code, including plenty of checks. | |
| · Set the random number seed once per simulation repetition. | |
| · Store the random number states at the start of each repetition. | |
| · If running chunks of the simulation in parallel, use separate streams of random numbers.17 | |
| analysis | 5 |
| · Conduct exploratory analysis of results, particularly graphical exploration. | |
| · Compute estimates of performance and Monte Carlo SEs for these estimates. | 5.2 |
| reporting | 6 |
| · Describe simulation study using ADEMP structure with sufficient rationale for choices. | |
| · Structure graphical and tabular presentations to place performance of competing methods side‐by‐side. | |
| · Include Monte Carlo SE as an estimate of simulation uncertainty. | 5.2 |
| · Publish code to execute the simulation study including user‐written routines. | 8 |