TABLE 1.
Examples of Diagnosands and the Elements of the Model (M), Inquiry (I), Data Strategy (D), and Answer Strategy (A) Required in Order for a Design to be Diagnosand-Complete for Each Diagnosand
| Required: |
|||||
|---|---|---|---|---|---|
| Diagnosand | Description | M | I | D | A |
| Power | Probability of rejecting null hypothesis of no effect | ✓ | ✓ | ✓ | |
| Estimation bias | Expected difference between estimate and estimand | ✓ | ✓ | ✓ | ✓ |
| Sampling bias | Expected difference between population average treatment effect and sample average treatment effect (Imai, King, and Stuart 2008) | ✓ | ✓ | ✓ | |
| RMSE | Root mean-squared-error | ✓ | ✓ | ✓ | ✓ |
| Coverage | Probability the confidence interval contains the estimand | ✓ | ✓ | ✓ | ✓ |
| SD of estimates | Standard deviation of estimates | ✓ | ✓ | ✓ | |
| SD of estimands | Standard deviation of estimands | ✓ | ✓ | ✓ | |
| Imbalance | Expected distance of covariates across treatment conditions | ✓ | ✓ | ||
| Type S rate | Probability estimate has incorrect sign, if statistically significant (Gelman and Carlin 2014) | ✓ | ✓ | ✓ | ✓ |
| Exaggeration ratio | Expected ratio of absolute value of estimate to estimand, if statistically significant (Gelman and Carlin 2014) | ✓ | ✓ | ✓ | ✓ |
| Value for money | Probability that a decision based on estimated effect yields net benefits | ✓ | ✓ | ✓ | ✓ |
| Robustness | Joint probability of rejecting the null hypothesis across multiple tests | ✓ | ✓ | ✓ | |