Table 2.
Reliability and accuracy statistics for 13 family therapy techniques presented in the training system, averaged across 32 weeks
| Family therapy Modules & techniques |
Average concordance: continuous |
Average concordance: dichotomous |
Average score: gold-standard |
Average score: therapist |
Equality of means | |
|---|---|---|---|---|---|---|
| ICC(1,2) | Cohen’s kappa | Sensitivitya %/specificityb% | M (SD) | M (SD) | t-test | |
| Family engagement | ||||||
| 1. Parent collaboration | 0.64 | 0.59 | 89/71 | 1.17 (1.36) | 1.28 (1.27) | 9.37*** |
| 2. Love and commitment | 0.65 | 0.66 | 98/62 | 1.69 (1.43) | 1.97 (1.27) | 9.93*** |
| 3. Parent ecosystem | 0.58 | 0.35 | 88/59 | 0.47 (0.78) | 1.08 (1.24) | 9.24*** |
| 4. Adolescent goal collab. | 0.40 | 0.34 | 81/53 | 1.75 (1.66) | 1.38 (1.24) | − 6.32*** |
| Relational orientation | ||||||
| 5. Relational focus | − 0.55 | – | 93/– | 3.39 (0.80) | 2.19 (1.09) | − 15.11*** |
| 6. Focus on process | 0.11 | 0.16 | 98/15 | 1.72 (1.32) | 2.34 (1.05) | 11.73*** |
| 7. Reframe | 0.30 | 0.13 | 88/27 | 1.90 (1.19) | 1.88 (1.13) | − .33 |
| 8. Relational reframe | 0.31 | 0.03 | 83/21 | 1.33 (1.27) | 1.82 (1.25) | 7.65*** |
| 9. Family-focused rationale | 0.58 | 0.33 | 88/44 | 1.58 (1.11) | 1.68 (1.22) | 1.68 |
| Interactional change | ||||||
| 10. Prepare for interactions | 0.83 | 0.39 | 87/58 | 1.08 (1.67) | 1.48 (1.54) | 3.83*** |
| 11. Stimulate dialogue | 0.67 | 0.32 | 99/30 | 1.63 (1.53) | 2.28 (1.33) | 9.51*** |
| 12. Coach and process | 0.69 | 0.69 | 95/72 | 1.50 (1.37) | 2.02 (1.54) | 6.82*** |
| 13. Teach family skills | 0.53 | 0.30 | 91/38 | 1.17 (1.17) | 1.78 (1.37) | 6.41*** |
| Family engagement score | – | – | – | 1.27 (1.23) | 1.37 (1.10) | 2.74** |
| Relational orientation score | – | – | – | 1.86 (1.00) | 1.98 (.94) | 3.45** |
| Interactional change score | – | – | – | 1.10 (1.16) | 1.72 (1.28) | 13.05*** |
| Scale total score | – | – | – | 1.59 (0.41) | 1.78 (0.73) | 9.88*** |
Bold values indicate data pertaining to an averaged scale score
Kappa was not calculated for Relational Focus due to 100% appearance as a target item (it never appeared as a contrast item)
*p < .05. **p < .01. ***p < .001
aRate of identifying true target items. Targets items were items that were depicted in the vignette and received a gold-standard score of 1–4. Each weekly vignette coding activity included three target items. Items were presented as target items 36–100% of total appearances (mean = 64% positive appearances)
bRate of identifying true contrast items. Contrast items were items that were not depicted in the vignette and received a gold-standard score of 0. Each weekly vignette coding activity contained two contrast items