Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2007 Oct 27.
Published in final edited form as: J Consult Clin Psychol. 2007 Oct;75(5):707–715. doi: 10.1037/0022-006X.75.5.707

Prediction of Suicide Ideation and Attempts Among Adolescents Using a Brief Performance-Based Test

Matthew K Nock 1, Mahzarin R Banaji 1
PMCID: PMC2043087  NIHMSID: NIHMS29245  PMID: 17907852

Abstract

Suicide is a leading cause of death that is difficult to predict because clinical assessment has relied almost exclusively on individuals' self-report of suicidal thoughts. This is problematic because there often is motivation to conceal such thoughts. The authors tested the ability of the Self-Injury Implicit Association Test (SI-IAT), a reaction-time measure of implicit associations between self-injury and oneself, to detect and predict suicide ideation and attempts. Participants were adolescents who were nonsuicidal (n = 38), suicide ideators (n = 37), or recent suicide attempters (n = 14). Analyses revealed large between-group differences on the SI-IAT, with nonsuicidal adolescents showing large negative associations between self-injury and themselves, suicide ideators showing small positive associations, and suicide attempters showing large positive associations on this performance-based test. The SI-IAT accurately predicted current suicide ideation and attempt status as well as future suicide ideation, and it incrementally improved prediction of these outcomes above and beyond the use of known risk factors. Future research is needed to refine this assessment method and to further develop and examine performance-based assessment of suicide risk in clinical settings.

Keywords: suicide, self-injury, implicit association test, assessment, prediction

Nearly 1 million people kill themselves worldwide each year, equaling one death by suicide approximately every 40 s (Goldsmith, Pellmar, Kleinman, & Bunney, 2002; World Health Organization, 2005). Despite decades of clinical, scientific, and policy efforts aimed at improving methods for predicting and preventing suicide, the rates of suicidal thoughts and attempts have remained virtually unchanged (Kessler, Berglund, Borges, Nock, & Wang, 2005). A persistent barrier encountered by clinicians is that current clinical assessment methods rely almost exclusively on self-report of suicidal thoughts and intentions.

This is problematic because suicidal individuals often conceal or deny such thoughts in order to avoid unwanted intervention efforts, such as involuntary hospitalization, or to facilitate release from such settings. Suicidal thoughts may go unreported for other reasons as well. For instance, sicidal thoughts typically are transient in nature and may be absent during clinical interview but then resurface shortly thereafter, such as following discharge from a secure psychiatric setting. Some individuals may even lack introspective awareness of the thoughts and feelings that drive suicidal behavior and thus lack the ability to inform others of their presence. Prior research has indicated that although 50%–69% of those who die by suicide communicate suicidal thoughts or intent to others in some way before they die (Coombs et al., 1992; Robins, Gassner, Kayes, Wilkinson, & Murphy, 1959), 78% of patients who die by suicide explicitly deny suicidal thoughts in their last communications before killing themselves (Busch, Fawcett, & Jacobs, 2003). Moreover, the risk of suicide death is significantly elevated immediately following hospital discharge, presumably shortly after patients denied suicidal intent (Goldacre, Seagroatt, & Hawton, 1993; Qin & Nordentoft, 2005). Overall, individuals who kill themselves shortly after denying suicidal thoughts and intent might (a) purposely conceal the presence of existing suicidal thoughts and intentions from clinicians, (b) fail to experience such thoughts during clinical assessment only to have them resurface shortly thereafter, or (c) lack conscious awareness of such thoughts. Whatever the reason in any particular case, it is clear that new clinical assessment methods are sorely needed that are not based solely on individuals' self-report of suicidal thoughts.

Cognitive and social scientists recently have developed indirect, performance-based methods of measuring individuals' implicit thoughts about various constructs in ways that do not rely on self-report (Fazio & Olson, 2003). The Implicit Association Test (IAT; Greenwald, McGhee, & Schwartz, 1998) is one such method used primarily to examine implicit associations people hold about nonclinical constructs such as racial prejudice (Olsson, Ebert, Banaji, & Phelps, 2005; Rudman, Ashmore, & Gary, 2001), gender stereotypes (Nosek, Banaji, & Greenwald, 2002), and ethical beliefs (Banaji, Bazerman, & Chugh, 2003). The IAT has several strengths that make it particularly well-suited for the assessment of psychopathology in general (Palfai & Wagner, 2004; Teachman, Gregg, & Woody, 2001) and of self-injury propensity in particular. It has been shown to have strong reliability (Cunningham, Preacher, & Banaji, 2001; Greenwald & Nosek, 2001), construct validity (Lane, Banaji, Nosek, & Greenwald, in press), and sensitivity to clinical change in treatment (Teachman & Woody, 2003), and perhaps most important it is resistant to attempts to “fake good” (Banse, Seise, & Zerbes, 2001).

The purpose of the current line of research is to translate the work of social and cognitive psychologists into a clinical assessment method that could be used to detect and predict self-injurious behavior without relying on explicit self-report. Toward this end, we have developed a self-injury IAT (SI-IAT) that measures the implicit associations individuals hold about self-injury. One version of the SI-IAT measures individuals' implicit identity with self-injury (i.e., the extent to which they associate self-injury with themselves) and a second version measures individuals' implicit attitude about self-injury (i.e., the extent to which they associate self-injury with being a favorable vs. unfavorable behavior). We recently demonstrated that the SI-IAT can improve the cross-sectional prediction of nonsuicidal self-injury (NSSI), with particularly strong effects for the implicit identity SI-IAT (Nock & Banaji, 2007). Here we extend this work by further examining the usefulness of the implicit identity SI-IAT. More specifically, we tested whether this performance-based test could also predict suicide ideation and suicide attempts, as well as whether it could do so prospectively, which would be most useful for clinical purposes. We examined suicide ideation and attempts in this report separately from NSSI because prior research has demonstrated the importance of distinguishing between suicidal and nonsuicidal self-injury and of considering and examining these constructs independently (Linehan, 1997; Muehlenkamp, 2005; Nock & Kessler, 2006).

In this study, we first tested whether performance on this brief, computerized, reaction-time test differs among nonsuicidal individuals, suicide ideators, and suicide attempters. We also tested whether the SI-IAT can distinguish among these suicide groups even after accounting for the presence of NSSI given our earlier findings (Nock & Banaji, 2007) and also because NSSI often co-occurs with suicide ideation and attempts (e.g., Nock, Joiner, Gordon, Lloyd-Richardson, & Prinstein, 2006). The demonstration of such differences would provide the first evidence of a performance-based measure that can distinguish between levels of suicide risk and that would support the SI-IAT's sensitivity in detecting these important group differences. Second, we tested the accuracy of the SI-IAT in statistically predicting current suicide ideation and suicide attempt status, as well as in prospectively predicting suicide ideation over the 6 months following administration of this test. Demonstrating that the SI-IAT can accurately predict these outcomes is a necessary and important step in examining the usefulness of this measure for the prospective prediction of suicide attempts and deaths, which is the ultimate and most important goal of this line of clinical research. Support for the ability of the SI-IAT to predict suicidal outcomes over time is especially important given the transient nature of suicidal thoughts and intentions. Our third and final test examined whether the SI-IAT adds incrementally to the prediction of suicide ideation and attempts above and beyond the use of well-known risk factors for these outcomes. Such a finding would show that this performance-based test can actually improve upon current methods for assessing and predicting these suicidal outcomes.

Method

Participants

Eighty-nine (68 female, 21 male) adolescents (age in years: M = 17.10, SD = 1.92, range = 12–19) participated in this study. We focused on adolescence given the significantly increased risk of suicidal thoughts and behaviors during this developmental period (Kessler, Borges, & Walters, 1999; Nock & Kazdin, 2002). The sample included nonsuicidal controls (n = 38), participants with a recent history (i.e., in the past year) of suicide ideation (n = 37), and those with a recent history (i.e., in the past year) of a suicide attempt (n = 14). This sample size provided fair to strong statistical power to detect the large between-group differences necessary for the SI-IAT to be a useful clinical tool (power = .71 to .93 for two-tailed tests with alpha set at .05). Participants in all conditions were recruited via announcements posted in local psychiatric clinics, in newspapers, on community bulletin boards, and on the Internet. The announcements for both control and self-injurious participants indicated the following: “We are seeking adolescents between the ages of 12 to 19, and their parents, to participate in a study aimed at understanding self-harm behaviors. Eligible participants will be paid for participation in this confidential study. Participation involves completing interviews, questionnaires, and computer tasks.” All procedures were approved by the university's institutional review board. Written informed consent was obtained for all participants, with parental consent obtained for those less than 18 years of age.

Assessment

Given that one of our goals was to test the incremental validity of the SI-IAT, in addition to suicide-related constructs, we assessed demographic and psychiatric factors shown in prior studies to predict suicide ideation and suicide attempts.

Demographic factors

Demographic factors including age, sex, and ethnicity were assessed in face-to-face interviews, given that they have been shown in prior work to be related to suicide ideation and attempts (American Academy of Child and Adolescent Psychiatry, 1997; American Psychiatric Association, 2003; Kessler et al., 1999; Nock & Kazdin, 2002). To ensure that between-group differences on the SI-IAT were not due to differences in IQ, all participants also were assessed using the Weschler Abbreviated Scales of Intelligence (Weschler, 1999).

DSM-IV disorders

The presence and number of psychiatric disorders have been shown to predict suicide ideation and attempts (American Academy of Child and Adolescent Psychiatry, 1997; American Psychiatric Association, 2003; Kessler et al., 1999; Nock & Kazdin, 2002) and were therefore assessed using the Schedule for Affective Disorders and Schizophrenia for School-Age Children—Present and Lifetime Version (K-SADS-PL; Kaufman, Birmaher, Brent, Rao, & Ryan, 1997). The K-SADS-PL is a semistructured diagnostic interview that assesses current and past episodes of 33 different psychiatric disorders according to the fourth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV; American Psychiatric Association, 1994). The K-SADS-PL was administered by the first author and four trained and supervised graduate research assistants. Independent ratings were completed for 20 randomly selected interviews and revealed strong interrater reliability (average κ = .93 across all diagnoses). Given their associations with suicide ideation and attempts, we focused specifically on disorders of mood (major depression, bipolar), anxiety (panic, separation anxiety, phobias, generalized anxiety, and obsessive-compulsive), impulse control (oppositional defiant, conduct, attention deficit/hyperactivity), eating (bulimia, anorexia), and substance use (alcohol, drugs).

Suicide ideation and suicide attempts

Suicide ideation and attempts were assessed using multiple methods. All participants were administered the Self-Injurious Thoughts and Behaviors Interview (SITBI; Nock, Holmberg, Photos, & Michel, in press), a structured clinical interview that assesses the presence, frequency, severity, age-of-onset, and other characteristics of a broad range of self-injurious thoughts and behaviors including suicide ideation and suicide attempts. The SITBI has strong interrater reliability (average κ = .99), test–retest reliability over a 6-month period (average κ = .70), and construct validity as demonstrated by strong relations with other measures of suicide ideation (average κ = .54) and suicide attempt (κ= .65; Nock et al., in press). Several study variables were derived from responses to the SITBI. First, participants were classified into one of the three mutually exclusive study groups on the basis of their responses to items regarding the presence of suicide ideation and attempts in the year preceding the baseline assessment (i.e., “Have you had thoughts of killing yourself in the past year?” “Have you made an actual attempt to kill yourself in the past year in which you had at least some intent to die?”). Second, given that past suicidal behavior has been shown to be the best predictor of future suicidal behavior (Joiner et al., 2005; Joiner & Rudd, 2000), we created variables of prior history of suicide ideation and suicide attempts (i.e., presence of each of these constructs at any time prior to the year preceding the baseline interview). Third, the SITBI was readministered by telephone 6 months after the baseline interview to assess the presence of suicide ideation and attempts in the 6 months following the baseline interview.

In addition to the SITBI, all participants completed the Beck Scale for Suicide Ideation (BSI; Beck & Steer, 1991), a 21-item self-report measure of the presence and severity of current suicide ideation. The BSI is a widely used measure of suicide ideation that has strong psychometric properties, which have been demonstrated in adult as well as adolescent samples (Allan, Kashani, Dahlmeier, Taghizadeh, & Reid, 1997; Nock & Kazdin, 2002). Scores on the BSI supported the suicide group classifications made using the SITBI, with nonsuicidal individuals reporting less suicide ideation (M = 1.1, SD = 2.6) than the suicide ideation (M = 5.8, SD = 5.9) and suicide attempt (M = 13.0, SD = 8.5) groups, F(2, 86) = 26.30, p < .001.

SI-IAT

The SI-IAT was developed, administered, and scored according to recommended IAT procedures (Greenwald, Nosek, & Banaji, 2003; Nosek, Greenwald, & Banaji, 2005). Participants sat alone at a desktop computer and were instructed to classify stimuli that appeared in the center of the computer screen as quickly as possible by pressing the following two corresponding keys: “e” for stimuli to be classified on the left of the screen and “i” for stimuli to be classified on the right (see https://implicit.harvard.edu/implicit/ for demonstration tests). The IAT rests on the assumption that it should be easier to make the same behavioral response (i.e., a key press) to concepts that are strongly associated relative to concepts that are weakly associated.

In the SI-IAT examined in this study, participants were presented with a series of images that are either self-injury related (i.e., pictures of skin that has been cut) or neutral (i.e., pictures of noninjured skin) and were asked to classify these as quickly as possible as representing the concepts “cutting” or “no cutting.” Although this focus on cutting is likely to also be relevant to individuals who engage in NSSI (Nock & Prinstein, 2004, 2005), we intentionally focused on this single and simple stimulus in this first test of the SI-IAT given that it is unambiguously related to self-injury (i.e., stimuli such as firearms and tall buildings are more complex and may not be perceived as self-injurious related even by many suicidal individuals) and thus limits confusion and variability in the test procedures. This decision also was made on the basis of concerns that have been raised about the potential iatrogenic effects of presenting adolescents who have a history of suicidal behavior with stimuli that are explicitly suicide-related (Shaffer et al., 1990). Participants also are presented with words that are either self-relevant (e.g., I, Mine) or other-relevant (e.g., They, Them) and are asked to classify these as quickly as possible as representing the attributes “me” or “not me.” Correct classifications are followed by the presentation of the next stimulus and incorrect classifications are followed by the presentation of a red “X” below the stimulus, which remains until the correct key press is made.

In the first critical test block (presented in random order), participants must press the same computer key in response to both “cutting” and “me” stimuli, and the other computer key for “no cutting” and “not me” stimuli. In the second critical test block, the opposite sorting is performed, pairing “cutting/not me” on the same computer key and “no cutting/me” on the other. Response latencies in these two blocks are recorded and analyzed using the most recently prescribed IAT scoring algorithm (Greenwald et al., 2003). The relative strength of the association between self-injury and oneself is indexed by calculating a D score for each participant by subtracting the average response latency of the “cutting/me” test block from the average response latency of the “cutting/not me” test block and dividing by the standard deviation of response latency for all trials. Thus, positive D scores represent relatively faster responding (i.e., stronger associations) when self-injury and oneself are paired, whereas negative D scores represent relatively slower responding (i.e., weaker associations) when self-injury and oneself are paired.

Procedures

Participants completed all of the measures described above during one baseline visit. Six months later, participants were contacted via telephone and were readministered the SITBI to evaluate the predictive validity of the SI-IAT. Follow-up data were obtained for 73 (82.0%) of the participants. Six participants could not be located, 7 did not respond to repeated requests for an interview, and 3 refused to participate in the follow-up interview. There were no significant differences between those who participated in follow-up interviews and those who did not on any of the key study variables: age, sex, ethnicity; presence or number of psychiatric disorders; presence of suicide ideation or attempts at the baseline interview; or score on the SI-IAT.

Data Analysis

To test the first study hypothesis, performance on the SI-IAT (i.e., D scores) among the three groups was compared using t tests for independent samples. Second, the ability of the SI-IAT to independently predict suicide ideation and suicide attempts was tested using separate logistic regression analyses for each suicide-related outcome, with regression coefficients converted to odds ratios with 95% confidence intervals for ease of interpretation. In addition, receiver operating characteristic curve analyses (Hsiao, Bartko, & Potter, 1989; Zweig & Campbell, 1993) were used to evaluate the ability of the SI-IAT to correctly classify suicide ideators and attempters. Third, the ability of the SI-IAT to add incrementally to the prediction of suicide ideation and suicide attempts was tested using separate hierarchical logistic regression analyses for each outcome. Consistent with prior recommendations on the statistical prediction of suicide-related outcomes (Cohen, 1986), significant demographic risk factors were entered in the first step, significant psychiatric risk factors in the second step, and scores on the SI-IAT in the third and final step. Analyses predicting 6-month suicide ideation also controlled for baseline scores on the BSI in order to conduct a more stringent test of the incremental predictive validity of the SI-IAT. All tests were two-tailed with alpha set at .05.

Results

Performance on the SI-IAT Among Suicide Groups

Analyses revealed large and statistically significant differences on the SI-IAT between nonsuicidal individuals and both suicide ideators, t(73) = −3.39, d = 0.78, p < .001, and suicide attempters, t(50) = −4.53, d = 1.28, p < .001, as well as between suicide ideators and suicide attempters, t(49) = −2.72, d = 0.78, p = .009. As presented in Figure 1, these large group differences were consistent with study hypotheses, with nonsuicidal individuals showing a negative association between self-injury and oneself (M = −0.25, SE = 0.07), suicide ideators showing a small positive association between self-injury and oneself (M = 0.06, SE = 0.06), and suicide attempters showing a large positive association between self-injury and oneself (M = 0.40, SE = 0.14).

Figure 1.

Figure 1

Open in a new tab

Differences in performance on the SI-IAT among suicide groups. Error bars represent standard error of the mean. All group differences are statistically significant (p < .05).

It is possible that the observed effects could be driven primarily by the presence of NSSI among participants in the suicide ideation and attempt groups. To test this, we conducted an analysis of suicide group differences among only those participants with a lifetime history of NSSI. Results were very similar to those reported above, with a negative association on the SI-IAT for non-suicidal individuals (n = 14; M = −0.09, SE = 0.15), a small positive association for suicide ideators (n = 36; M = 0.09, SE = 0.06), and a strong positive association for suicide attempters (n = 14; M = 0.41, SE = 0.14), F(2, 58) = 4.45, p < .05. These results suggest that the differences among suicide groups are not entirely due to the presence of NSSI.

It also is possible that the observed effects were driven primarily by suicide attempters who used cutting as a suicide attempt method and that those who used other methods (e.g., drug overdose) would show weaker associations on the SI-IAT. Analyses suggested that this also was not the case, as there were no differences in SI-IAT performance between suicide attempters who had never used cutting as a method of suicide attempt (n = 10; M = 0.40, SE = 0.16) and those who had (n = 4; M = 0.42, SE = 0.33), t(12) = −0.06, ns.

Predictive Validity of the SI-IAT

Performance on the SI-IAT was strongly and consistently predictive of recent suicide ideation and attempts, as well as subsequent suicide ideation at 6-month follow-up, with higher scores on the SI-IAT associated with significantly greater odds of each suicidal outcome, as presented in Table 1. The area under the receiver operating characteristic curve for each suicidal outcome was .74 –.77. This means that a randomly selected suicidal individual (as defined by each analysis) could be distinguished from a randomly selected nonsuicidal individual with 74%–77% accuracy based on performance on the SI-IAT, demonstrating a relatively high level of precision for this performance-based test.

Table 1.

Predictive Validity of the SI-IAT

Variable OR (95% CI) Model χ2(1) AUC (95% CI) p
Suicide ideation 10.63 (3.08, 36.68) 19.10 .77 (0.66, 0.87) <.001
Suicide attempt 10.91 (2.66, 45.86) 13.44 .76 (0.61, 0.91) <.001
6-month suicide ideation 7.50 (2.09, 26.98) 11.52 .74 (0.63, 0.85) .001
Open in a new tab

Note. SI-IAT = Self-injury Implicit Association Test; OR = odds ratio; CI = confidence interval; AUC = area under the receiver operating characteristic curve.

Incremental Predictive Validity of the SI-IAT

The final study hypothesis was that the SI-IAT would improve the prediction of suicide ideation and attempts above and beyond the effect of demographic and psychiatric factors. The three suicide groups did not differ on sex, ethnicity, IQ, or the presence of any anxiety, impulse-control, or eating disorder, as presented in Table 2. There was a small but statistically significant between-groups difference on age as well as larger group differences on the presence of any mood and substance use disorder, total number of psychiatric disorders, and presence of prior suicide ideation and attempt. Therefore, each of these variables was statistically controlled in subsequent analyses (Cohen, 1986).

Table 2.

Characteristics of the Three Participant Groups

Nonsuicidal Suicide ideators Suicide attempters
Variable (n = 38) (n = 37) (n = 14) Test p
Age in years, M ± SD 16.9 ± 1.9 17.6 ± 1.6 16.2 ± 2.2 F(2, 86) = 3.37   .039
Sex (% male) 26.3 24.3 14.3 χ2(2) = 0.84 .657
Ethnicity (%)
 European American 65.8 75.7 85.7 χ2(10) = 5.33   .868
 African American  2.6  5.4  0.0
 Hispanic  7.9  5.4  7.1
 Asian  7.9  2.7  0.0
 Biracial 13.2 10.8  7.1
 Other  2.6  0.0  0.0
Full-scale IQ, M ± SD 110.9 ± 11.3 108.9 ± 12.0 109.4 ± 12.5 F(2, 86) = 0.23  .795
DSM-IV diagnoses (%)
 Any mood disorder  7.9 40.5 85.7   χ2(2) = 29.05 <.001
 Any anxiety disorder 36.8 54.1 57.1 χ2(2) = 2.89 .236
 Any impulse-control disorder 10.5 13.5 14.3 χ2(2) = 0.21 .900
 Any eating disorder  7.9  2.7 14.3 χ2(2) = 2.31 .315
 Any substance use disorder  2.6 27.0 14.3 χ2(2) = 8.95 .011
Number of DSM-IV disorders, M ± SD  0.9 ± 1.6  1.9 ± 2.0  2.6 ± 1.5 F(2, 86) = 5.20   .007
Prior suicide ideation (%) 31.6 73.0 78.6   χ2(2) = 16.43 <.001
Prior suicide attempt (%)  7.9 21.6 50.0   χ2(2) = 11.31 .003
Open in a new tab

Note. DSM-IV = Diagnostic and Statistical Manual of Mental Disorders (4th edition).

Hierarchical logistic regression analyses revealed that, after accounting for the variance explained by significant demographic and psychiatric risk factors, performance on the SI-IAT predicted significant unique variance in suicide ideation (Table 3) and suicide attempts (Table 4). Moreover, in prospective analyses over the 6-month follow-up period, performance on the SI-IAT predicted the occurrence of suicide ideation even after controlling for demographic and psychiatric risk factors, including prior history of suicide ideation and attempts and baseline presence and severity of suicide ideation as measured by the BSI (Table 5). These findings indicate that the SI-IAT improves prediction of these outcomes above and beyond the influence of known demographic and psychiatric risk factors for these outcomes.

Table 3.

Hierarchical Logistic Regression Analyses Predicting Recent (Past Year) Suicide Ideation

Variable B SE Wald OR (95% CI) χ2 R2
Step 1 χ2(1) = 0.42 .01
 Age 0.07 0.11 0.42 1.08 (0.86, 1.34) 
Step 2 χ2(5) = 38.64*** .48
 Any mood disorder 2.72 0.89 9.47 15.22 (2.69, 86.27)**
 Any substance use disorder 3.36 1.40 5.72 28.73 (1.83, 450.64)*
 Total number of disorders −0.34 0.23 2.17 0.71 (0.45, 1.12) 
 Prior suicide ideation 1.77 0.61 8.50 5.85 (1.78, 19.16)*
 Prior suicide attempt 0.44 0.75 0.34 1.55 (0.36, 6.69) 
Step 3 χ2(1) = 5.77* .53
 SI-IAT 1.62 0.77 5.12 5.07 (1.24, 20.71)*
Open in a new tab

Note. SI-IAT = Self-Injury Implicit Association Test; OR = odds ratio; CI = confidence interval.

*

p < .05.

**

p < .01.

***

p < .001.

Table 4.

Hierarchical Logistic Regression Analyses Predicting Recent (Past Year) Suicide Attempt

Variable B SE Wald OR (95% CI) χ2 R2
Step 1 χ2(1) = 3.32 .06
 Age −0.26 0.14 3.39 0.77 (0.58, 1.02)  
Step 2 χ2(5) = 33.97*** .59
 Any mood disorder 4.78 1.50 10.13 119.46 (6.29, 2270.34)***
 Any substance use disorder 0.12 1.49 0.01 1.12 (0.61, 20.55)  
 Total number of disorders −0.40 0.39 1.02 0.67 (0.31, 1.45)  
 Prior suicide ideation 1.40 1.23 1.30 4.07 (0.36, 45.54)  
 Prior suicide attempt 2.23 1.11 4.03 9.25 (1.05, 81.29)* 
Step 3 χ2(1) = 7.83** .68
 SI-IAT 2.62 1.10 5.71 13.70 (1.60, 117.37)* 
Open in a new tab

Note. SI-IAT = Self-Injury Implicit Association Test; OR = odds ratio; CI = confidence interval.

*

p < .05.

**

p < .01.

***

p < .001.

Table 5.

Hierarchical Logistic Regression Analyses Predicting Prospective (6-Month) Suicide Ideation

Variable B SE Wald OR (95% CI) χ2 R2
Step 1 χ2(1) = 0.00 .00
 Age −0.01 0.13 0.00 0.99 (0.76, 1.29)
Step 2 χ2(6) = 19.64** .34
 Any mood disorder 0.27 0.91 0.09 1.31 (0.22, 7.93)
 Any substance use disorder −1.06 1.07 0.98 0.34 (0.04, 2.83)
 Total number of disorders −0.11 0.23 0.23 0.90 (0.57, 1.40)
 Prior suicide ideation 0.86 0.71 1.48 2.37 (0.59, 9.57)
 Prior suicide attempt −0.27 0.86 0.10 0.76 (0.14, 4.12)
 BSI 0.70 0.25 7.78 2.00 (1.23, 3.27)*
Step 3 χ2(1) = 5.48* .42
 SI-IAT 1.86 0.86 4.66 6.41 (1.19, 34.58)*
Open in a new tab

Note. BSI = Beck Scale for Suicide Ideation; SI-IAT = Self-Injury Implicit Association Test; OR = odds ratio; CI = confidence interval.

*

p < .05.

**

p < .01.

Preliminary Evidence for Predicting Prospective Suicide Attempts

Two participants made a suicide attempt during the 6-month follow-up period. For exploratory purposes, we examined the performance of these participants on the SI-IAT and observed that their scores (M = 0.71, SE = 0.51) were significantly higher than those who did not make a suicide attempt during the follow-up period (M = 0.00, SE = 0.05), t(71) = 2.18, d = .52, p = .032. Although statistically significant, these results must be interpreted with caution given that they are based on the performance of only two suicide attempters. Nevertheless, these findings provide preliminary evidence that the SI-IAT may be useful in the prospective prediction of suicide attempts.

Discussion

Suicide ideation and attempts are notoriously difficult to predict and prevent due to the concealment of suicidal thoughts, as well as the transient nature of such thoughts and the limitations in introspective access to the mental precursors of suicide. These problems have remained largely unaddressed in both clinical and research settings because of the lack of an alternative to using individuals' self-report of their thoughts about self-injury. This study provides the first evidence for a performance-based test of self-injurious thoughts that can be used to improve the assessment of suicide risk. The SI-IAT revealed large differences among nonsuicidal individuals, suicide ideators, and suicide attempters. The large and significant difference on the SI-IAT between suicide ideators and attempters is especially striking given the relatively limited range of constructs shown to distinguish between those who think about suicide and those who make an actual suicide attempt (Brent et al., 1988; Kessler et al., 1999; Nock & Kazdin, 2002; Nock & Kessler, 2006).

The fact that these differences were apparent even after accounting for the presence of NSSI suggests that although the stimuli used portrayed skin cutting, this test may be useful for suicide prediction beyond its relation to NSSI (Nock & Banaji, 2007). This may be because skin cutting is not just a method of NSSI but also is a common method of attempting suicide and thus is relevant to suicide ideators and attempters regardless of NSSI history. Moreover, the images of skin cutting clearly represent the general construct of self-injury and thus seem to be a useful stimulus set even for suicidal individuals who have never made a suicide attempt using cutting as a method. However, it is very important to note that many of the suicide ideators and all of the suicide attempters included in this sample also had a history of NSSI, the majority of whom used cutting as a method of NSSI, and so these results may not generalize to those with no history of NSSI. It is important for future research to examine performance on the SI-IAT among suicide attempters with no history of cutting of any kind to determine the generality of this new test.

The SI-IAT predicted suicidal outcomes both concurrently and prospectively with relatively strong precision. Indeed, the accuracy of the SI-IAT (area under the receiver operating characteristic curve = .74–.77) was at a level similar to that reported in related areas of clinical science such as the prediction of violence (Loeber et al., 2005; Mossman, 1994; Swets, Dawes, & Monahan, 2000). The SI-IAT also added incrementally to the prediction of suicidal outcomes beyond the use of several commonly used risk factors, including demographic factors and the presence of mood disorders, substance use, a prior history of suicide ideation and attempts, and baseline level of self-reported suicide ideation. Overall, these results suggest that the SI-IAT holds promise as a new behavioral method of detecting and predicting suicidal thoughts and behaviors.

Beyond improving clinical assessment and prediction of suicidal outcomes, these findings provide important information about the cognitions held by people who experience suicide ideation and attempts. Prior work has identified factors that increase the risk of suicidal outcomes, such as the presence of mental disorders; however, surprisingly little research has addressed the more specific question of how suicidal individuals actually think about self-injury or what associations they hold about such behaviors. Our results suggest that suicide ideators and attempters implicitly identify with self-injury. This identification with self-injury may be an important (and perhaps necessary) occurrence in the development of the ability to make a suicide attempt. Indeed, most people who experience depression and hopelessness never make a suicide attempt, and this may be because they do not think of self-injury as being within their behavioral repertoire. It may be that as individuals think about suicide, they develop a slight identification with self-injury, and as this identification strengthens they become more likely and more able to make an actual suicide attempt (see Joiner, 2005). The findings from this study (e.g., see Figure 1) support such a model. Of course, a plausible alternative explanation for these data is that an implicit identification with self-injury is a consequence of suicide ideation and attempts. In such a case, implicit identification with self-injury would be a marker for future suicidal outcomes rather than a cause of them. Prospective and treatment studies are needed to better understand the nature of the relation between implicit identification with self-injury and suicide ideation and attempts revealed in this study.

The findings from this study must be interpreted in the context of several important limitations. First, this test of the SI-IAT used a relatively small and self-selected sample. Many of the participants had a lifetime history of NSSI, and so it is important to replicate these findings among suicidal individuals with no history of NSSI in order to determine the generality of the results obtained. Also, although the sample size was sufficient to test the performance of the SI-IAT in distinguishing between nonsuicidal individuals, suicidal ideators, and suicide attempters, there were relatively large confidence intervals around the coefficients in the hierarchical logistic regression models, and there were only two suicide attempts over the 6-month follow-up period, limiting our ability to conduct an adequate test of the prospective prediction of suicide attempts. Given that the ultimate goal of this line of work is to improve prediction of suicide attempts and suicide death in clinical settings, it is important that future studies of the SI-IAT examine its performance among larger samples of individuals presenting at clinical settings, such as psychiatric emergency departments, inpatient units, and outpatient clinics.

Second, as mentioned above, this initial SI-IAT used concepts and stimuli representing only one form of self-injury (i.e., cutting) rather than the broad and heterogeneous range of possible suicide methods, such as the use of a firearm, jumping from heights, drug overdose, and the like. In addition, the stimuli did not assess participants' intent to die from self-injury. On balance, this limitation is actually likely to have suppressed the predictive ability of this SI-IAT, as using stimuli more directly related to suicide probably would have generated even stronger results. Of course, despite this limitation, use of the current stimuli accurately predicted suicide-related outcomes. The promise of the results obtained here, along with recent findings showing that the presentation of suicide-related items does not necessarily increase distress and suicide risk (Gould et al., 2005), provides justification for extending the repertory of concepts and stimuli to include other methods of self-injury as well as stimuli related more directly to concepts such as death and suicide. Future versions of the SI-IAT will incorporate these distinctions, and doing so is likely to enhance the ability of this test to more successfully predict suicide ideation and attempts.

Third, although the sample size provided sufficient statistical power to detect the large effects observed in this initial study, we were not able to examine potentially important relations between performance on the SI-IAT and more specific aspects of suicide attempts. For instance, prior work has suggested that those who make multiple suicide attempts are a more severe and higher risk group than those who have made only a single suicide attempt and also that important aspects of the experience of suicidal thoughts and crises differ between single and multiple attempters (Joiner & Rudd, 2000). Examining the relation between the performance on the SI-IAT and single versus multiple suicide attempt status, as well as other characteristics of suicide attempts such as lethality and intent, remain essential directions for future research in this area.

These limitations notwithstanding, the results of this study have significant implications for scientific and clinical work on the prediction and prevention of suicide. Future research in this area should incorporate the modifications outlined above, replicate these findings in larger and higher risk samples in which there are likely to be greater numbers of suicide attempts, and refine the administration and scoring procedures in order to further increase the accuracy and feasibility of this method as a useful clinical assessment tool. Beyond clinical prediction, alternative versions of the SI-IAT, such as those measuring implicit cognitions about death, despair, foreshortened future, and other constructs, can be used to advance scientific understanding of the psychological experiences of suicidal individuals. Similarly, the SI-IAT can be used by treatment researchers to measure implicit cognitions over the course of an intervention (Teachman & Woody, 2003), providing valuable information for testing hypothesized mechanisms of clinical change (Kazdin & Nock, 2003; Nock, in press).

Although a substantial gap exists between what is feasible in the laboratory versus the clinic in many areas of science, the SI-IAT provides a viable and valuable opportunity to translate findings from basic behavioral research in a way that directly informs prediction and decision making in clinical settings (Swets et al., 2000; Zerhouni, 2005). The addition of implicit measurement methods to clinical interviews, self-report, and biological assessment tools (e.g., Mann et al., 2006) may significantly improve the ability of scientists and clinicians to predict and prevent the loss of life due to suicide and represents a new and exciting direction for clinical research.

Acknowledgments

This research was supported by National Institute of Mental Health Grants MH076047 (M.K.N.) and MH57672 (M.R.B.), as well as by grants from the Milton Fund and Talley Fund of Harvard University (M.K.N.). We thank members of the Laboratory for Clinical and Developmental Research and Roy Ruhling for their assistance with this work.

References

  1. Allan WD, Kashani JH, Dahlmeier J, Taghizadeh P, Reid JC. Psychometric properties and clinical utility of the scale for suicide ideation with inpatient children. Journal of Abnormal Child Psychology. 1997;25:465–473. doi: 10.1023/a:1022633714888. [DOI] [PubMed] [Google Scholar]
  2. American Academy of Child and Adolescent Psychiatry Practice parameters for the assessment and treatment of children and adolescents with conduct disorder. Journal of the American Academy of Child and Adolescent Psychiatry. 1997;36S:122S–139S. doi: 10.1097/00004583-199710001-00008. [DOI] [PubMed] [Google Scholar]
  3. American Psychiatric Association . Diagnostic and statistical manual of mental disorders. 4th ed. Author; Washington, DC: 1994. [Google Scholar]
  4. American Psychiatric Association Practice guideline for the assessment and treatment of patients with suicidal behaviors. American Journal of Psychiatry. 2003;160(Suppl 11):1–60. [PubMed] [Google Scholar]
  5. Banaji MR, Bazerman MH, Chugh D. How (un)ethical are you? Harvard Business Review. 2003;81:56–64. 125. [PubMed] [Google Scholar]
  6. Banse R, Seise J, Zerbes N. Implicit attitudes towards homosexuality: Reliability, validity, and controllability of the IAT. Zeitschrift Fur Experimentelle Psychologie. 2001;48:145–160. doi: 10.1026//0949-3946.48.2.145. [DOI] [PubMed] [Google Scholar]
  7. Beck AT, Steer RA. Manual for the Beck Scale for Suicide Ideation. Psychological Corporation; San Antonio, TX: 1991. [Google Scholar]
  8. Brent DA, Perper JA, Goldstein CE, Kolko DJ, Allan MJ, Allman CJ, et al. Risk factors for adolescent suicide. A comparison of adolescent suicide victims with suicidal inpatients. Archives of General Psychiatry. 1988;45:581–588. doi: 10.1001/archpsyc.1988.01800300079011. [DOI] [PubMed] [Google Scholar]
  9. Busch KA, Fawcett J, Jacobs DG. Clinical correlates of inpatient suicide. Journal of Clinical Psychiatry. 2003;64:14–19. doi: 10.4088/jcp.v64n0105. [DOI] [PubMed] [Google Scholar]
  10. Cohen J. Statistical approaches to suicidal risk factor analysis. Annals of the New York Academy of Sciences. 1986;487:34–41. doi: 10.1111/j.1749-6632.1986.tb27883.x. [DOI] [PubMed] [Google Scholar]
  11. Coombs DW, Miller HL, Alarcon R, Herlihy C, Lee JM, Morrison DP. Presuicide attempt communications between parasuicides and consulted caregivers. Suicide and Life-Threatening Behavior. 1992;22:289–302. [PubMed] [Google Scholar]
  12. Cunningham WA, Preacher KJ, Banaji MR. Implicit attitude measures: Consistency, stability, and convergent validity. Psychological Science. 2001;12:163–170. doi: 10.1111/1467-9280.00328. [DOI] [PubMed] [Google Scholar]
  13. Fazio RH, Olson MA. Implicit measures in social cognition research: Their meaning and use. Annual Review of Psychology. 2003;54:297–327. doi: 10.1146/annurev.psych.54.101601.145225. [DOI] [PubMed] [Google Scholar]
  14. Goldacre M, Seagroatt V, Hawton K. Suicide after discharge from psychiatric inpatient care. Lancet. 1993;342:283–286. doi: 10.1016/0140-6736(93)91822-4. [DOI] [PubMed] [Google Scholar]
  15. Goldsmith SK, Pellmar TC, Kleinman AM, Bunney WE, editors. Reducing suicide: A national imperative. The National Academies Press; Washington, DC: 2002. [PubMed] [Google Scholar]
  16. Gould MS, Marrocco FA, Kleinman M, Thomas JG, Mostkoff K, Cote J, et al. Evaluating iatrogenic risk of youth suicide screening programs: A randomized controlled trial. Journal of the American Medical Association. 2005;293:1635–1643. doi: 10.1001/jama.293.13.1635. [DOI] [PubMed] [Google Scholar]
  17. Greenwald AG, McGhee DE, Schwartz JL. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology. 1998;74:1464–1480. doi: 10.1037//0022-3514.74.6.1464. [DOI] [PubMed] [Google Scholar]
  18. Greenwald AG, Nosek BA. Health of the Implicit Association Test at age 3. Zeitschrift Fur Experimentelle Psychologie. 2001;48:85–93. doi: 10.1026//0949-3946.48.2.85. [DOI] [PubMed] [Google Scholar]
  19. Greenwald AG, Nosek BA, Banaji MR. Understanding and using the implicit association test: I. An improved scoring algorithm. Journal of Personality and Social Psychology. 2003;85:197–216. doi: 10.1037/0022-3514.85.2.197. [DOI] [PubMed] [Google Scholar]
  20. Hsiao JK, Bartko JJ, Potter WZ. Diagnosing diagnoses. Receiver operating characteristic methods and psychiatry. Archives of General Psychiatry. 1989;46:664–667. doi: 10.1001/archpsyc.1989.01810070090014. [DOI] [PubMed] [Google Scholar]
  21. Joiner TE. Why people die by suicide. Harvard University Press; Cambridge, MA: 2005. [Google Scholar]
  22. Joiner TE, Conwell Y, Fitzpatrick KK, Witte TK, Schmidt NB, Berlim MT, et al. Four studies on how past and current suicidality relate even when “everything but the kitchen sink” is covaried. Journal of Abnormal Psychology. 2005;114:291–303. doi: 10.1037/0021-843X.114.2.291. [DOI] [PubMed] [Google Scholar]
  23. Joiner TE, Rudd MD. Intensity and duration of suicidal crises vary as a function of previous suicide attempts and negative life events. Journal of Consulting and Clinical Psychology. 2000;68:909–916. doi: 10.1037/0022-006X.68.5.909. [DOI] [PubMed] [Google Scholar]
  24. Kaufman J, Birmaher B, Brent DA, Rao U, Ryan ND. Schedule for Affective Disorders and Schizophrenia for School-Age Children—Present and Lifetime Version (K-SADS-PL): Initial reliability and validity data. Journal of the American Academy of Child and Adolescent Psychiatry. 1997;36:980–988. doi: 10.1097/00004583-199707000-00021. [DOI] [PubMed] [Google Scholar]
  25. Kazdin AE, Nock MK. Delineating mechanisms of change in child and adolescent therapy: Methodological issues and research recommendations. Journal of Child Psychology and Psychiatry. 2003;44:1116–1129. doi: 10.1111/1469-7610.00195. [DOI] [PubMed] [Google Scholar]
  26. Kessler RC, Berglund P, Borges G, Nock MK, Wang PS. Trends in suicide ideation, plans, gestures, and attempts in the United States, 1990–1992 to 2001–2003. Journal of the American Medical Association. 2005;293:2487–2495. doi: 10.1001/jama.293.20.2487. [DOI] [PubMed] [Google Scholar]
  27. Kessler RC, Borges G, Walters EE. Prevalence of and risk factors for lifetime suicide attempts in the National Comorbidity Survey. Archives of General Psychiatry. 1999;56:617–626. doi: 10.1001/archpsyc.56.7.617. [DOI] [PubMed] [Google Scholar]
  28. Lane K, Banaji MR, Nosek BA, Greenwald AG. Understanding and using the Implicit Association Test: IV. What we know (so far) about the method. In: Wittenbrink B, Schwarz N, editors. Implicit measures of attitudes: Procedures and controversies. Guilford Press; New York: in press. [Google Scholar]
  29. Linehan MM. Behavioral treatments of suicidal behaviors. Definitional obfuscation and treatment outcomes. Annals of the New York Academy of Sciences. 1997;836:302–328. doi: 10.1111/j.1749-6632.1997.tb52367.x. [DOI] [PubMed] [Google Scholar]
  30. Loeber R, Pardini D, Homish DL, Wei EH, Crawford AM, Farrington DP, et al. The prediction of violence and homicide in young men. Journal of Consulting and Clinical Psychology. 2005;73:1074–1088. doi: 10.1037/0022-006X.73.6.1074. [DOI] [PubMed] [Google Scholar]
  31. Mann JJ, Currier D, Stanley B, Oquendo MA, Amsel LV, Ellis SP. Can biological tests assist prediction of suicide in mood disorders? International Journal of Neuropsychopharmacology. 2006;9:465–474. doi: 10.1017/S1461145705005687. [DOI] [PubMed] [Google Scholar]
  32. Mossman D. Assessing predictions of violence: Being accurate about accuracy. Journal of Consulting and Clinical Psychology. 1994;62:783–792. doi: 10.1037//0022-006x.62.4.783. [DOI] [PubMed] [Google Scholar]
  33. Muehlenkamp JJ. Self-injurious behavior as a separate clinical syndrome. American Journal of Orthopsychiatry. 2005;75:324–333. doi: 10.1037/0002-9432.75.2.324. [DOI] [PubMed] [Google Scholar]
  34. Nock MK. Conceptual and design essentials for evaluating mechanisms of change. Alcoholism: Clinical and Experimental Research. doi: 10.1111/j.1530-0277.2007.00488.x. in press. [DOI] [PubMed] [Google Scholar]
  35. Nock MK, Banaji MR. Assessment of self-injurious thoughts using a behavioral test. American Journal of Psychiatry. 2007;164:820–823. doi: 10.1176/ajp.2007.164.5.820. [DOI] [PubMed] [Google Scholar]
  36. Nock MK, Holmberg EB, Photos VI, Michel BD. Self-Injurious Thoughts and Behaviors Interview: Development, reliability, and validity in an adolescent sample. Psychological Assessment. doi: 10.1037/1040-3590.19.3.309. in press. [DOI] [PubMed] [Google Scholar]
  37. Nock MK, Joiner TE, Jr., Gordon KH, Lloyd-Richardson E, Prinstein MJ. Non-suicidal self-injury among adolescents: Diagnostic correlates and relation to suicide attempts. Psychiatry Research. 2006;144:65–72. doi: 10.1016/j.psychres.2006.05.010. [DOI] [PubMed] [Google Scholar]
  38. Nock MK, Kazdin AE. Examination of affective, cognitive, and behavioral factors and suicide-related outcomes in children and young adolescents. Journal of Clinical Child and Adolescent Psychology. 2002;31:48–58. doi: 10.1207/S15374424JCCP3101_07. [DOI] [PubMed] [Google Scholar]
  39. Nock MK, Kessler RC. Prevalence of and risk factors for suicide attempts versus suicide gestures: Analysis of the National Comorbidity Survey. Journal of Abnormal Psychology. 2006;115:616–623. doi: 10.1037/0021-843X.115.3.616. [DOI] [PubMed] [Google Scholar]
  40. Nock MK, Prinstein MJ. A functional approach to the assessment of self-mutilative behavior. Journal of Consulting and Clinical Psychology. 2004;72:885–890. doi: 10.1037/0022-006X.72.5.885. [DOI] [PubMed] [Google Scholar]
  41. Nock MK, Prinstein MJ. Clinical features and behavioral functions of adolescent self-mutilation. Journal of Abnormal Psychology. 2005;114:140–146. doi: 10.1037/0021-843X.114.1.140. [DOI] [PubMed] [Google Scholar]
  42. Nosek BA, Banaji MR, Greenwald AG. Math = male, me = female, therefore math not = me. Journal of Personality and Social Psychology. 2002;83:44–59. [PubMed] [Google Scholar]
  43. Nosek BA, Greenwald AG, Banaji MR. Understanding and using the Implicit Association Test: II. Method variables and construct validity. Personality and Social Psychology Bulletin. 2005;31:166–180. doi: 10.1177/0146167204271418. [DOI] [PubMed] [Google Scholar]
  44. Olsson A, Ebert JP, Banaji MR, Phelps EA. The role of social groups in the persistence of learned fear. Science. 2005 July;309:785–787. doi: 10.1126/science.1113551. [DOI] [PubMed] [Google Scholar]
  45. Palfai T, Wagner EF. Current perspectives on implicit cognitive processing in clinical disorders: Implications for assessment and intervention [Introduction to special series] Cognitive and Behavioral Practice. 2004;11:135–138. [Google Scholar]
  46. Qin P, Nordentoft M. Suicide risk in relation to psychiatric hospitalization: Evidence based on longitudinal registers. Archives of General Psychiatry. 2005;62:427–432. doi: 10.1001/archpsyc.62.4.427. [DOI] [PubMed] [Google Scholar]
  47. Robins E, Gassner S, Kayes J, Wilkinson RH, Jr., Murphy GE. The communication of suicidal intent: A study of 134 consecutive cases of successful (completed) suicide. American Journal of Psychiatry. 1959;115:724–733. doi: 10.1176/ajp.115.8.724. [DOI] [PubMed] [Google Scholar]
  48. Rudman LA, Ashmore RD, Gary ML. “Unlearning” automatic biases: The malleability of implicit prejudice and stereotypes. Journal of Personality and Social Psychology. 2001;81:856–868. doi: 10.1037//0022-3514.81.5.856. [DOI] [PubMed] [Google Scholar]
  49. Shaffer D, Vieland V, Garland A, Rojas M, Underwood M, Busner C. Adolescent suicide attempters. Response to suicide-prevention programs. Journal of the American Medical Association. 1990;264:3151–3155. [PubMed] [Google Scholar]
  50. Swets JA, Dawes RM, Monahan J. Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest. 2000;1:1–26. doi: 10.1111/1529-1006.001. [DOI] [PubMed] [Google Scholar]
  51. Teachman BA, Gregg AP, Woody SR. Implicit associations for fear-relevant stimuli among individuals with snake and spider fears. Journal of Abnormal Psychology. 2001;110:226–235. doi: 10.1037//0021-843x.110.2.226. [DOI] [PubMed] [Google Scholar]
  52. Teachman BA, Woody SR. Automatic processing in spider phobia: Implicit fear associations over the course of treatment. Journal of Abnormal Psychology. 2003;112:100–109. [PubMed] [Google Scholar]
  53. Weschler D. Wechsler Abbreviated Scale of Intelligence. Psychological Corporation; San Antonio, TX: 1999. [Google Scholar]
  54. World Health Organization World Health Organization: Suicide Prevention. 2005 Retrieved October 26, 2005, from http://www.who.int/mental_health/prevention/suicide/suicideprevent/en/
  55. Zerhouni EA. Translational and clinical science—time for a new vision. New England Journal of Medicine. 2005;353:1621–1623. doi: 10.1056/NEJMsb053723. [DOI] [PubMed] [Google Scholar]
  56. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry. 1993;39:561–577. [PubMed] [Google Scholar]

RESOURCES