TABLE 2.
Results of inter‐rater reliability exercises: Chatter characteristics and behaviors
| Variable | Proportion agreement observed | Proportion agreement expected | Kappa | Maximum kappa | ||||
|---|---|---|---|---|---|---|---|---|
| Time 1 a | Time 3 a | Time 1 | Time 3 | Time 1 | Time 3 | Time 1 | Time 3 | |
| Demographics | ||||||||
| LGBTQ | 1 | 1 | .680 | .936 | 1 | 1 | 1 | 1 |
| Military status | .978 | .978 | .956 | .956 | .489 | .489 | .489 | .489 |
| Prior chats | .956 | 1 | .894 | b | .580 | b | .580 | b |
| Life stressors | ||||||||
| Break‐up | .978 | .978 | .785 | .708 | .896 | .924 | .896 | .924 |
| Isolation | .889 | .844 | .680 | .563 | .653 | .644 | .653 | .797 |
| Sexual abuse | 1 | 1 | b | b | b | b | b | b |
| Bullying | 1 | 1 | .936 | b | 1 | b | 1 | b |
| Other abuse | .933 | .956 | .895 | .737 | .365 | .831 | .365 | .831 |
| Relationship issue | .822 | .800 | .580 | .506 | .577 | .595 | .577 | .730 |
| Death | .978 | .956 | .895 | .803 | .788 | .775 | .788 | .775 |
| Other's suicide | .933 | .956 | .838 | .838 | .588 | .726 | .588 | .726 |
| Financial problem | .933 | .889 | .737 | .680 | .746 | .653 | .916 | .653 |
| Barrier to treat. | .978 | .911 | .957 | .737 | .489 | .662 | .489 | .662 |
| Insurance prob. | 1 | .978 | .936 | .857 | 1 | .845 | 1 | .845 |
| Lost job | .911 | .956 | .838 | .820 | .451 | .753 | .863 | .753 |
| Job stress | .956 | .844 | .838 | .753 | .726 | .371 | .863 | .640 |
| School problem | .978 | .933 | .708 | .753 | .924 | .730 | .924 | .910 |
| Gender/Sexuality | .978 | 1 | .895 | .936 | .788 | 1 | .788 | 1 |
| Physical problem | .978 | .911 | .857 | .803 | .845 | .550 | .845 | .663 |
| Homelessness | .978 | .978 | .978 | .915 | −.011 | .738 | −.011 | .738 |
| Suicide risk factors | ||||||||
| Lifetime ideation | .956 | .956 | .680 | .680 | .861 | .861 | 1 | 1 |
| Current ideation | .811 | .889 | .508 | .454 | .616 | .797 | .864 | .837 |
| Imminent risk | .765 | .875 | .569 | .605 | .454 | .684 | .545 | .789 |
| Passive ideation | .833 | .667 | .500 | .605 | .667 | .330 | 1 | .598 |
| Attempt in prog. | .978 | 1 | .978 | b | −.011 | b | −.011 | b |
| Prep for future | .972 | 1 | .973 | b | −.014 | b | −.014 | b |
| Prep for past | .944 | .942 | .946 | .818 | −.021 | .682 | .234 | .841 |
| Method chosen | .917 | .913 | .500 | .500 | .833 | .826 | .833 | .884 |
| Gun | 1 | .978 | b | .857 | b | .845 | b | .845 |
| Hanging | 1 | 1 | b | .936 | b | 1 | b | 1 |
| Pills | 1 | 1 | .510 | .680 | 1 | 1 | 1 | 1 |
| Gas | 1 | 1 | .755 | b | 1 | b | 1 | b |
| Drowning | 1 | 1 | .867 | b | 1 | b | 1 | b |
| Bridge/Height | 1 | 1 | b | .936 | b | 1 | b | 1 |
| Moving object | 1 | 1 | .867 | b | 1 | b | 1 | b |
| Knife/Sharp | 1 | 1 | .592 | .876 | 1 | 1 | 1 | 1 |
| Means available | .758 | c | .341 | c | .632 | c | .816 | c |
| Place chosen | .889 | 1 | .871 | b | .140 | b | .355 | b |
| Said had plan | .917 | .913 | .556 | .397 | .812 | .856 | .812 | .856 |
| Prior attempt(s) | .917 | .884 | .463 | .580 | .845 | .724 | .922 | .724 |
| NSSI | .911 | .911 | .511 | .609 | .818 | .773 | .864 | .773 |
| Additional suicide risk factors | ||||||||
| Substance abuse | .878 | .956 | .762 | .895 | .486 | .577 | .486 | .577 |
| Intoxicated | 1 | 1 | .936 | b | 1 | b | 1 | b |
| SA tx (ever) | 1 | 1 | b | b | b | b | b | b |
| SA tx (current) | – | 1 | – | b | – | b | – | b |
| MH diagnosis | .911 | .889 | .583 | .571 | .787 | .741 | .893 | .793 |
| MH tx (ever) | .956 | .911 | .411 | .449 | .925 | .839 | .925 | .839 |
| MH tx (current) | .867 | .846 | .380 | .396 | .785 | .746 | .929 | .767 |
| Psych. hosp. (ever) | .978 | .933 | .753 | .876 | .910 | .464 | .910 | .643 |
| Recent discharge | .978 | 1 | .915 | b | .738 | b | .738 | b |
| Reasons for living | .889 | .840 | .639 | .706 | .693 | .457 | .754 | .638 |
| Steps agreed upon by chatter to mitigate imminent risk | ||||||||
| Emergency rescue | 1 | 1 | b | b | b | b | b | b |
| Self‐rescue | 1 | 1 | b | b | b | b | b | b |
| Involve others now | 1 | 1 | .803 | b | 1 | b | 1 | b |
| Move to phone | 1 | 1 | b | b | b | b | b | b |
| Remove means | 1 | 1 | b | .936 | b | 1 | b | 1 |
| Follow‐up call | 1 | 1 | b | b | b | b | b | b |
| Risk reduced | 1 | 1 | .625 | c | 1 | 1 | 1 | 1 |
| Chatter behavior at end of chat | ||||||||
| Said thanks | .889 | .933 | .509 | .502 | .774 | .866 | 1 | .911 |
| Felt better | .844 | .889 | .563 | .556 | .644 | .750 | .898 | .850 |
| Accepted coping | .844 | .911 | .694 | .803 | .492 | .550 | .928 | .775 |
| Accepted MH tx | .933 | .911 | .838 | .753 | .589 | .640 | .863 | .820 |
| Felt sleepy | .978 | .956 | .915 | .936 | .738 | .310 | .738 | .310 |
| Had to go | .911 | .933 | .694 | .753 | .710 | .730 | .910 | .730 |
| Apologetic | .911 | .956 | .857 | .956 | .380 | −.023 | .845 | −.023 |
| Dissatisfied | .978 | .956 | .785 | .936 | .896 | .310 | .896 | .310 |
| Ending too soon | 1 | 1 | b | b | b | b | b | b |
| Abandoned chat | .978 | .933 | .915 | .753 | .738 | .730 | .738 | .910 |
| Chatter's behavioral changes by end of chat | ||||||||
| Less overwhelmed | .711 | .911 | .609 | .838 | .261 | .451 | .432 | .863 |
| More hopeful | .800 | .800 | .708 | .753 | .316 | .191 | .544 | .391 |
| More confident | .800 | .844 | .667 | .667 | .400 | .533 | .800 | .666 |
| Less at risk d | .767 | .833 | .442 | .489 | .582 | .674 | .841 | .783 |
a
Inter‐rater reliability exercises, each involving 30 chats coded by three coders, were conducted at the beginning, middle, and end of transcript coding. Results of the first and third exercises are presented here. Results of the second exercise were consistent with these and are available upon request.
b
Coders agreed perfectly on these items; however, kappa could not be calculated due to a lack of variability in responses. This could happen, for example, if all coders answered “No” for all chats.
c
Too few cases to calculate reliability.
d
Dichotomized as “Not at all” vs. “A little” or “Moderately/A lot”.