Table 5.
Prevalence and magnitude of response shift results for different study design characteristics
| Study design characteristics | Study-level results | Effect-level results | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Total effects | Recalibration effects | Reprioritization and/or Reconceptualization effects, or unknowna | ||||||||||
| Prevalenceb | Prevalenceb | Prevalenceb | Magnitudec | Prevalenceb | Magnitudec | |||||||
| N | % RS detected | N | % RS detected | N | % RS detected | N | Median ES (IQR) | N | % RS detected | N | Median ES (IQR) | |
| Design | ||||||||||||
| Observational | 122 | 88.5 | 3978 | 16.2 | 1724 | 28.1 | 879 | 0.21 (0.10–0.36) | 2254 | 7.1 | 33 | 0.10 (0.04–0.17) |
| Experimental | 28 | 78.6 | 890 | 16.5 | 361 | 27.7 | 145 | 0.26 (0.16–0.41) | 529 | 8.9 | 5 | 0.03 (0.01–0.06) |
| Data analysis | ||||||||||||
| Primary analysis | 90 | 87.8 | 1590 | 30.2 | 1058 | 39.7 | 872 | 0.22 (0.11–0.38) | 532 | 11.3 | 15 | 0.14 (0.08–0.25) |
| Secondary analysis | 59 | 84.7 | 3268 | 9.5 | 1027 | 16.0 | 152 | 0.19 (0.09–0.32) | 2241 | 6.6 | 23 | 0.05 (0.02–0.10) |
| Unknown | 1 | 100.0 | 10 | 10.0 | n/a | n/a | n/a | n/a | 10 | 10.0 | n/a | n/a |
| Sample sizes (Binned) | ||||||||||||
| Q1 (< 57) | 59 | 76.3 | 1121 | 16.1 | 584 | 23.3 | 391 | 0.26 (0.13–0.45) | 537 | 8.4 | 5 | 0.03 (0.01–0.06) |
| Q2 (57–254) | 79 | 82.3 | 1353 | 23.4 | 668 | 35.5 | 393 | 0.22 (0.10–0.38) | 685 | 11.5 | 13 | 0.14 (0.10–0.20) |
| Q3 (255–410) | 34 | 85.3 | 1262 | 15.8 | 472 | 32.0 | 194 | 0.15 (0.09–0.24) | 790 | 6.2 | 8 | 0.10 (0.07–0.35) |
| Q4 (> 411) | 27 | 77.8 | 1132 | 8.4 | 361 | 16.6 | 46 | 0.20 (0.12–0.35) | 771 | 4.5 | 12 | 0.05 (0.02–0.13) |
| Time period classification | ||||||||||||
| < 1 month | 15 | 80.0 | 488 | 13.1 | 205 | 24.9 | 95 | 0.23 (0.09–0.32) | 283 | 4.6 | 7 | 0.01 (0.00–0.03) |
| 1–6 months | 23 | 91.3 | 427 | 19.9 | 110 | 43.6 | 49 | 0.40 (0.22–0.71) | 317 | 11.7 | 11 | 0.05 (0.04–0.10) |
| > 6–12 months | 36 | 72.2 | 905 | 12.3 | 347 | 21.0 | 78 | 0.18 (0.09–0.39) | 558 | 6.8 | n/a | n/a |
| > 12 months | 90 | 87.8 | 2604 | 18.2 | 1211 | 29.7 | 733 | 0.22 (0.11–0.38) | 1393 | 8.1 | 16 | 0.13 (0.09–0.28) |
| Not reported | 12 | 83.3 | 444 | 13.3 | 212 | 24.5 | 69 | 0.16 (0.10–0.28) | 232 | 3.0 | 4 | 0.18 (0.13–0.24) |
aUnknown includes several effects for which the pathway was unknown due to it not being explicitly reported.
bPrevalence is based on a total of 150 studies and 4868 effects that were investigated or possible. N = # of studies or response shift effects. The # of studies may not add up to 150 (first column) because some studies implemented multiple methods or had multiple samples that were counted separately. % RS detected = the percentage of detected response shift effects of # studies conducted or # response shift effects that were investigated or possible.
cMagnitude is based on 91 studies and 1062 effects for which Cohen’s d could be determined. N = the # of these response shift effects. ES = Cohen’s d. IQR = interquartile range. n/a = not applicable because either the response shift pathway could not be investigated due to the method used or the results could not be discerned from what was reported in the manuscript.
Example: For observational study designs, there are a total of 122 studies of which 88.5% had detected response shift and 3978 effects of which 16.2% had detected response shift. There were 1724 recalibration effects, of which 28.1% had detected response shift and 879 for which a Cohen’s d was obtained. There were also 2254 reprioritization and/or reconceptualization or unknown effects, of which 7.1% had detected response shift and 33 for which a Cohen’s d was obtained.