TABLE 3.
Results of inter‐rater reliability exercises: Counselor behaviors
| Variable | Proportion observed | Proportion expected | Kappa | Maximum kappa | ||||
|---|---|---|---|---|---|---|---|---|
| Time 1 a | Time 2 a | Time 1 | Time 2 | Time 1 | Time 2 | Time 1 | Time 2 | |
| Fostering engagement/Rapport | ||||||||
| Welcome | .967 | .867 | .545 | .580 | .927 | .683 | .927 | .841 |
| Empathy | .900 | .967 | .745 | .905 | .608 | .649 | .608 | .649 |
| Strengths | .867 | .900 | .680 | .567 | .583 | .769 | .583 | .923 |
| Collaborative problem‐solving | ||||||||
| Past skills | .867 | .967 | .536 | .625 | .713 | .911 | 1 | .911 |
| Support solutions | .933 | 1 | .509 | .502 | .864 | 1 | 1 | 1 |
| Offer suggestions | .867 | .833 | .520 | .501 | .722 | .666 | .722 | .933 |
| Review plan | 1 | .933 | .936 | .876 | 1 | .464 | 1 | 1 |
| Suicide risk assessment | ||||||||
| Current thoughts? | .900 | .967 | .625 | .625 | .733 | .911 | .733 | .911 |
| Prior thoughts? | .967 | .900 | .625 | .745 | .911 | .608 | .911 | .608 |
| Past attempt? | .967 | 1 | .745 | .680 | .869 | 1 | .869 | 1 |
| Thoughts about method? | 1 | .933 | .609 | .502 | 1 | .866 | 1 | 1 |
| Available means? | 1 | 1 | .820 | .680 | 1 | 1 | 1 | 1 |
| Attempt in progress? | 1 | 1 | .609 | .642 | 1 | 1 | 1 | 1 |
| Intent to act? | 1 | .967 | .820 | .847 | 1 | .782 | 1 | .782 |
| Steps suggested by counselor to establish safety & mitigate risk | ||||||||
| Safety planning | 1 | 1 | .936 | b | 1 | b | 1 | b |
| Remove means | 1 | 1 | .936 | .936 | 1 | 1 | 1 | 1 |
| Move to phone | 1 | 1 | b | b | b | b | b | b |
| Rescue | 1 | .967 | .820 | .794 | 1 | .838 | 1 | .838 |
| Assess safety | 1 | .900 | .722 | .701 | 1 | .666 | 1 | .666 |
| Resources, referrals, & mental health treatment promotion | ||||||||
| Open to MH treatment? | .967 | .900 | .505 | .505 | .933 | .798 | .933 | .933 |
| Ways to access treatment | 1 | .967 | .820 | .905 | 1 | .649 | 1 | .649 |
| Referral | .967 | 1 | .967 | b | −.017 | b | −.017 | b |
| Other resources | 1 | .900 | .642 | .701 | 1 | .666 | 1 | .666 |
a
Inter‐rater reliability exercises, each involving 30 chats coded by two coders, were conducted at the beginning and end of transcript coding. Results of both exercises are presented here.
b
Coders agreed perfectly on these items; however, kappa could not be calculated due to a lack of variability in responses. This could happen, for example, if all coders answered “No” for all chats.