Reliability, Validity, and Clinical Utility of the Dominic Interactive for Adolescents–Revised

Abstract

Objectives:

The Dominic Interactive for Adolescents–Revised (DIA-R) is a multimedia self-report screen for 9 mental disorders, borderline personality traits, and suicidality defined by the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5). This study aimed to examine the reliability and the validity of this instrument.

Methods:

French- and English-speaking adolescents aged 12 to 15 years (N = 447) were recruited from schools and clinical settings in Montreal and were evaluated twice. The internal consistency was estimated by Cronbach alpha coefficients and the test-retest reliability by intraclass correlation coefficients. Cutoff points on the DIA-R scales were determined by using clinically relevant measures for defining external validation criteria: the Schedule for Affective Disorders and Schizophrenia for School-Aged Children, the Beck Hopelessness Scale, and the Abbreviated-Diagnostic Interview for Borderlines. Receiver operating characteristic (ROC) analyses provided accuracy estimates (area under the ROC curve, sensitivity, specificity, likelihood ratio) to evaluate the ability of the DIA-R scales to predict external criteria.

Results:

For most of the DIA-R scales, reliability coefficients were excellent or moderate. High or moderate accuracy estimates from ROC analyses demonstrated the ability of the DIA-R thresholds to predict psychopathological conditions. These thresholds were generally capable to discriminate between clinical and school subsamples. However, the validity of the obsessions/compulsions scale was too low.

Conclusions:

Findings clearly support the reliability and the validity of the DIA-R. This instrument may be useful to assess a wide range of adolescents’ mental health problems in the continuum of services. This conclusion applies to all scales, except the obsessions/compulsions one.

From a developmental psychopathology perspective, the biological and psychological changes occurring during the life transition period of adolescence may increase the vulnerability of youth to mental health problems.^1–4 The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5)⁵ supports epidemiological data suggesting that adolescents aged 12 to 15 years constitute a high-risk group for mental disorders,^6–14 which represent potential risk factors for suicidality.^15–21 In agreement with epidemiological^22–26 and clinical^27–29 studies, the DSM-5 recognizes the existence of borderline traits in adolescence.⁵ In this context, it is important to early identify DSM-based symptoms/criteria defining personality traits^4,27,29–33 as well as mental disorders^{1–3,6–14,16,34–36} and suicidal ideations/suicide attempts^{15–21,34–38} to prevent lifetime psychiatric disorders^11,12,30 and ultimately completed suicide.^{16–21,36–38}

Since no single informant can be considered the “gold standard,”^39–42 it is relevant to include an adolescent’s DSM-based self-report to provide unique, hence useful, information in a multi-informant evaluation process of youth’s psychopathology.^43,44 Various structured or semistructured diagnostic interviews were designed to assess a large number of DSM diagnoses according to adolescent report.^40,42 However, as we elaborated elsewhere,^43,44 their long administration time (60-120 minutes) and high level of complexity, particularly for adolescents aged 12 to 13 years, preclude their use by frontline service providers. From a prevention perspective, the development of an age-appropriate self-report screening instrument assessing DSM-5 symptoms for disorders,^43,44 borderline personality traits,³¹ and suicidality,^18,37 in the continuum of youth mental health services, remains a major challenge for clinicians and researchers.

Development of the Dominic Interactive for Adolescents–Revised

Printed^45–47 and computerized⁴⁸ versions of the Dominic were initially designed as age appropriate, that is, developmentally⁴⁹ sensitive pictorial self-reports of DSM symptoms for the most frequent disorders in children aged 6 to 11 years. Psychometric properties and the utility of the computerized Dominic Interactive (DI) were demonstrated in children from Canada,⁴⁸ the United States,⁵⁰ and European countries.^51–53

The development of a pictorial computerized self-report for adolescents is more recent.⁴³ Published in 2010, the Dominic Interactive for Adolescents (DIA) is a highly structured, multimedia self-report screen that uses visual (pictures, questions) and auditory stimuli (a voiceover asking questions) to assess current Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR)⁵⁴ symptoms defining 6 mental disorders: major depressive disorder (MDD), generalized anxiety disorder (GAD), specific phobia (SPh), attention-deficit/hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and conduct disorder (CD).⁴³ A 3-category rating system (likely absent, possible, or likely present) for each symptom scale indicates the tendency toward DSM disorders. We found adequate reliability and validity for the French version of the DIA in a sample of 607 adolescents aged 12 to 15 years from clinical and school populations.⁴³ The validity of the DIA was evaluated using 2 criteria: the clinical judgement on the presence/absence of symptoms (scored independently by 3 judges) and the adolescents’ referrals to outpatient psychiatric clinics. All symptom scales were valid, except separation anxiety,⁴³ which was excluded in the final version of the DIA. Moreover, since the DIA does not evaluate the frequency and duration of DSM criteria, changes about these dimensions proposed in the DSM-5 do not affect the DIA.

These previous works led to the development of a revised version, the Dominic Interactive for Adolescents–Revised (DIA-R), which adds 5 new DSM-5–based symptom scales: social phobia (SoPh), obsessive-compulsive disorder (OCD), substance use disorder (SUD), borderline traits (BT), and suicidality. The aim was to provide a technologically innovative self-report screen, capable of assessing a wider range of adolescents’ mental health problems within a brief administration time and suitable in frontline and psychiatric services.

Objectives

Using a convenience sample of French- and English-speaking Canadian adolescents aged 12 to 15 years, we addressed questions about fundamental steps in the validation process of the DIA-R.

The first objective was to analyze the reliability of the DIA-R scales (internal consistency, test-retest estimate of reliability)^55,56 according to age (12-13 years, 14-15 years), sex, language (French, English), and type of subsample (school, clinical).

The second and main objective was to examine the criterion-related validity of the 11 DIA-R scales to determine cutoff points by using clinically relevant measures⁴³ for defining external criteria: 1) the Schedule for Affective Disorders and Schizophrenia for School-Aged Children (KIDDIE-SADS) for 9 DIA-R symptom scales; 2) the suicidality (suicide plan/attempt) assessed by the KIDDIE-SADS, and this criterion combined with the level of hopelessness evaluated by the Beck Hopelessness Scale (BHS), for the DIA-R Suicidality Scale; and 3) the Abbreviated-Diagnostic Interview for Borderlines (Ab-DIB), as well as a combination of this instrument with the suicidality confirmed by the KIDDIE-SADS, for the DIA-R BT Scale.

The third objective was to verify the capacity of the DIA-R thresholds to discriminate between school and clinical subsamples. It represents another essential condition to demonstrate the usefulness of this instrument as a self-report screen.^42,47

Methods

Participants

The sampling plan aimed at recruiting a school subsample (n = 200) and a clinical subsample (n = 300) for obtaining accurate estimates for reliability and validity. Based on an expected reliability of 0.60 or higher, a sample size of 200 generates a standard error (SE) ≤0.05 or a confidence interval not wider than 0.20 for the internal consistency estimated by the Cronbach alpha coefficient^57,58 and the test-retest reliability estimated by the intraclass correlation coefficients (ICCs).⁵⁹ As for the validity estimated by the area under the receiver operating characteristic (ROC) curve (AUC), a sample size of 500 with at least 30 positive cases (at least 10% of clinical participants for each disorder) generates a SE ≤0.05 when the expected AUC is ≥0.80.⁶⁰

Visually or hearing-impaired adolescents and those with severe intellectual or learning disabilities were excluded. The final sample comprised 447 adolescents living in the Montreal urban area: 243 adolescents (130 French speaking, 113 English speaking) selected in regular classrooms from 4 secondary schools reflecting heterogeneous socioeconomic levels and 204 adolescents (171 French speaking, 33 English speaking) from youth centres, specialised psychiatric clinics, inpatient units, and day treatment centres. In these clinical settings, exclusion criteria were verified by the professionals responsible of the recruitment (e.g., nurses, psychologists, psychoeducators). The school subsample was balanced according to age, but boys (55.6%) and French-speaking adolescents (53%) were lightly overrepresented. In the clinical subsample, adolescents aged 12 to 13 years (29%) and English-speaking adolescents (16%) were underrepresented, while girls were lightly overrepresented (57%).

The institutional review boards of the participating hospitals and youth centres approved the research protocol (including the parent consent and the adolescent assent forms) for the clinical subsample while the review board of the principal investigator approved the protocol for the school subsample.

The DIA-R: Content and Structure

The DIA-R consists of 121 items from which 11 symptom/criteria scales are scored: MDD (19), GAD (16), SPh (7), SoPh (8), OCD (6), ADHD (18), ODD (9), CD (15), SUD (6), BT (9), and Suicidality (9).

The dimensional scales include a number of DSM-5 symptoms or phobic situations for SPh. The BT Scale is defined by an algorithm based on the 9 DSM-5 criteria⁵ describing personality traits (excluding the 1-year duration): fear of abandonment/unstable relationships, identity/self-image, impulsivity, and affective/dissociative symptoms. For the Suicidality Scale, a positive answer to 1 of the 2 main questions about suicidal ideation (e.g., “Do you often think about death or killing yourself?”) generates contingent questions regarding recurrent thoughts and plans in the past few weeks, as well as suicide attempts in the lifetime (e.g., “Have you ever tried to kill yourself?”) and in the past few weeks. Additionally, this scale includes 1 question about the “communication of suicidal intent”¹⁸ (e.g., “Did you ever tell anybody you might kill yourself?”). However, it does not assess other relevant components of explicit intent to die (e.g., prevention of discovery).^18,38 For examples of items, see Appendix I (available online).

The format of the DIA-R is similar to the initial DIA.⁴³ Although the duration of symptoms is not systematically evaluated, words such as often or most of the time are used to convey the recurrence of thoughts, emotions, or behaviours. The colour pictures present Dominic according to respondents’ sex and ethnicity (Caucasian, Hispanic, African American, or Asian). Adolescents disclose their responses by clicking the “Yes” or “No” boxes on the screen. Most adolescents complete the DIA-R within 20 to 25 minutes.

Measures of External Validation Criteria Used to Determine the DIA-R Cutoff Points

The KIDDIE-SADS is a semistructured diagnostic interview⁶¹ widely used in child psychiatry.^40,62 High levels of interrater agreement (93%-100%) and test-retest reliability (κ: 0.63-0.90) were found for the most frequent disorders (present time) in youth aged 7 to 17 years. The criterion-related validity was verified by using various external criteria.⁶¹ In our study, the English⁶¹ and French³⁶ versions of the KIDDIE-SADS were used to assess the adolescents’ perception of their symptoms for 9 disorders over the past 6 months (MDD, SPh, SoPh, GAD, OCD, ODD) or the past 12 months (ADHD, CD, SUD). The presence of a suicide plan/attempt during the past 6 months was retained as the main criterion to validate the Suicidality Scale included in the DIA-R. Changes proposed by the DSM-5 were taken into account during interviews (e.g., symptoms of ADHD prior to age 12 years) and were integrated into algorithms used to define mental disorders.

The BHS is a 20-item true/false self-report of negative expectations about the future experienced over the past week.^63–66 A BHS score ≥9 was combined with the presence of a suicide plan/attempt in the past week assessed by the KIDDIE-SADS. This second criterion was used to verify the capacity of the DIA-R Suicidality Scale to predict a more recent and severe suicidal risk. Studies using mainly adolescents aged 14 to 17 years highlighted the internal consistency of English (0.86)⁶⁶ and French (0.82)³⁵ versions of the BHS and an association between a higher level of hopelessness and suicidal ideations.^34,35,66 Our preliminary analyses confirmed acceptable internal consistency (≥0.81) regardless of age (12-13 years, 14-15 years), sex, and language subgroups.

The Ab-DIB is a 26-item self-report available in English and French for assessing the borderline traits over the past year.²⁷ The scores range from 0 to 52. Adequate psychometric properties were found in samples of suicidal youth aged 14 to 17 years.²⁷ Reliability coefficients were ≥0.80. Compared to the Diagnostic Interview for Borderlines–Revised,⁶⁷ the Ab-DIB demonstrated an AUC of 0.87. The highest efficiency cutoff was 11.50 (sensitivity: 0.88; specificity: 0.82).²⁷ Although the Ab-DIB was previously used in older adolescents,²⁷ our preliminary analyses revealed alpha coefficients ≥0.80 in all age (12-13 years, 14-15 years), sex, and language subgroups. To determine cutoff points on the DIA-R BT Scale, we used 2 criteria: 1) an Ab-DIB score ≥24 and 2) this score and the presence of a suicide plan/attempt as assessed by the KIDDIE-SADS. Since the Ab-DIB was used as an external validation criterion, we retained the score of 24 to increase its specificity and reduce false positives. Percentages of participants reaching this threshold were also more acceptable in the clinical (20%) and school (4%) subsamples.

Procedures

Prior to the data collection, interviewers received a 3-day training session on the content and the administration of the DIA-R, the BHS, and the Ab-DIB. The KIDDIE-SADS training extended over 1 month and covered theoretical aspects (e.g., correspondence with the DSM-5 symptoms/criteria), administration, and practice sessions. Interviewers had at least a bachelor’s degree in psychology or psychoeducation.

Adolescents were individually evaluated twice, 7 to 15 days apart (mean = 9.5, SD = 3.28). Half completed the DIA-R, the BHS, and the Ab-DIB at the first interview and the DIA-R and the KIDDIE-SADS at the second interview. The inverse order was proposed to the other adolescents. In the school subsample, both interviews took place at the adolescents’ school during the regular school hours. In the clinical subsample, interviews took place at the hospital or the youth centre. Most adolescents accepted an audio recording of the KIDDIE-SADS interviews. The principal investigator listened to 30% of the clinical recordings and provided feedback to each interviewer. However, this study did not aim to verify the interrater reliability of the KIDDIE-SADS.

Mental health professionals in various settings were informed of suicidal risks based on adolescents’ responses to questionnaires and intervened accordingly.

Statistical Analyses

Reliability

The internal consistency of the DIA-R was evaluated by Cronbach alpha coefficients^57,58 and the test-retest reliability by ICCs.⁵⁹ According to interpretative guidelines proposed by Shrout,^55,56 ICCs ≥0.60 were considered acceptable levels of reliability: moderate (0.61-0.80) and substantial/excellent (0.81-1.0). Alpha coefficients ≥0.70 suggested an acceptable internal consistency,⁶⁸ but for symptom scales including a smaller number of items (<12), the minimum adequacy criterion was around 0.60.⁶⁹

Criterion-related validity

The ROC analysis^60,70–73 provided accuracy estimates (AUC, sensitivity, specificity, likelihood ratio) to evaluate the ability of the 11 DIA-R scales to predict external validation criteria defining psychopathological conditions. The AUC is a global measure regarding the performance of a scale. It corresponds to the probability that a randomly chosen adolescent with a given psychopathology (e.g., MDD assessed by the KIDDIE-SADS) will have a higher score on a corresponding DIA-R scale than a randomly chosen adolescent without this condition. Guidelines applied in biomedical domains⁷² were suggested in psychiatry⁷¹ to interpret AUC values: high accuracy (>0.90), moderate (0.70-0.90), low (<0.70), and chance-level accuracy (0.50). In studies comparing dimensional scales with diagnostic interviews of youth, moderate accuracy (0.70-0.80) was generally found.⁷³ In our study, to minimize overreliance on memory of earlier responses, for all ROC analyses, when the external criterion was evaluated at the test, we used data from the DIA-R at retest (and vice versa).

For every DIA-R score, we calculated sensitivity (ability to detect youth with psychopathology), specificity (ability to detect youth without psychopathology), false positive (FP: 1 – specificity), and false negative (FN: 1 – sensitivity). Since this instrument may be used in different clinical and research settings, 2 cutoff points were determined on each scale: 1) a higher cutoff with high specificity to reduce FP (our targeted minimal specificity was 0.90) and 2) a lower cutoff with high sensitivity to reduce FN (our targeted minimal sensitivity was 0.75). Among all cutoff points respecting our a priori targets, final cutoff points were chosen to reach a balance between sensitivity and specificity.

The likelihood ratio positive (LR+) was calculated for all cutoff points on the DIA-R scales. Each LR+ indicates how more likely adolescents with a given psychopathology will yield a score equal to or above the cutoff points compared to those without this condition.

Finally, using Pearson’s χ² test, we compared school and clinical subsamples for the categories based on the DIA-R cutoff points.

Results

Internal Consistency

For the total sample, Cronbach alpha coefficients were >0.80 for MDD, Suicidality, SoPh, and ADHD scales and ≥0.85 for combined anxiety, internalizing, and externalizing symptoms. Coefficients were ≥0.75 for ODD, CD, SUD, GAD, and BT Scales. Coefficients were <0.60 for SPh (0.58) and OCD (0.48). The range of coefficients remained quite constant across age, sex, language, and type of subsample (school, clinical).

Test-Retest Estimate of Reliability

For the total sample (Table 1), ICCs ranged from 0.75 to 0.94 for specific scales and from 0.84 to 0.87 for the combined symptom scales. There was no major variation across age, sex, language, and type of subsample (school, clinical).

Table 1.

Test-Retest Reliability for DSM-5–Based Symptom Scales of the DIA-R According to Age, Sex, Language, School, and Clinical Subsamples (n = 431).

	ICC (95% CI)
DIA-R Scales	Total (n = 431)	12-13 Years (n = 177)	14-15 Years (n = 254)	Girls (n = 212)	Boys (n = 219)	French (n = 289)	English (n = 142)	School Subsample (n = 240)	Clinical Subsample (n = 191)
MDD	0.83 (0.80-0.86)	0.84 (0.78-0.88)	0.82 (0.77-0.85)	0.83 (0.78-0.86)	0.81 (0.75-0.85)	0.83 (0.80-0.87)	0.82 (0.76-0.87)	0.77 (0.72-0.82)	0.84 (0.79-0.87)
GAD	0.82 (0.78-0.85)	0.82 (0.76-0.86)	0.81 (0.77-0.85)	0.78 (0.72-0.83)	0.80 (0.75-0.85)	0.81 (0.76-0.84)	0.84 (0.78-0.88)	0.81 (0.76-0.85)	0.81 (0.75-0.85)
SPh	0.88 (0.86-0.90)	0.88 (0.85-0.91)	0.88 (0.85-0.91)	0.87 (0.84-0.90)	0.83 (0.78-0.87)	0.88 (0.85-0.90)	0.90 (0.87-0.93)	0.89 (0.86-0.91)	0.88 (0.84-0.91)
SoPh	0.86 (0.83-0.88)	0.87 (0.83-0.90)	0.84 (0.80-0.88)	0.85 (0.81-0.88)	0.83 (0.78-0.86)	0.86 (0.83-0.89)	0.84 (0.79-0.88)	0.81 (0.76-0.85)	0.89 (0.85-0.91)
OCD	0.79 (0.76-0.83)	0.83 (0.77-0.87)	0.77 (0.71-0.81)	0.79 (0.73-0.84)	0.78 (0.72-0.83)	0.78 (0.73-0.82)	0.81 (0.74-0.86)	0.79 (0.74-0.83)	0.80 (0.74-0.85)
ADHD	0.84 (0.81-0.87)	0.85 (0.81-0.89)	0.83 (0.79-0.87)	0.88 (0.84-0.90)	0.81 (0.76-0.85)	0.85 (0.81-0.88)	0.83 (0.77-0.87)	0.84 (0.80-0.87)	0.82 (0.77-0.86)
ODD	0.75 (0.71-0.79)	0.74 (0.66-0.80)	0.75 (0.69-0.80)	0.76 (0.70-0.82)	0.74 (0.67-0.79)	0.73 (0.67-0.78)	0.80 (0.73-0.85)	0.75 (0.69-0.80)	0.69 (0.60-0.76)
CD	0.84 (0.81-0.87)	0.82 (0.77-0.87)	0.85 (0.81-0.88)	0.83 (0.79-0.87)	0.85 (0.81-0.88)	0.82 (0.78-0.86)	0.88 (0.84-0.91)	0.73 (0.67-0.79)	0.84 (0.79-0.88)
SUD	0.94 (0.93-0.95)	0.93 (0.91-0.95)	0.94 (0.92-0.95)	0.96 (0.95-0.97)	0.92 (0.90-0.94)	0.94 (0.92-0.95)	0.95 (0.93-0.96)	0.90 (0.88-0.92)	0.93 (0.91-0.95)
BTa	0.77 (0.73-0.81)	0.76 (0.68-0.81)	0.77 (0.71-0.81)	0.76 (0.70-0.82)	0.76 (0.70-0.81)	0.78 (0.73-0.82)	0.77 (0.69-0.83)	0.74 (0.68-0.79)	0.75 (0.68-0.83)
Suicidality	0.75 (0.71-0.79)	0.71 (0.63-0.77)	0.77 (0.71-0.81)	0.74 (0.68-0.80)	0.76 (0.69-0.81)	0.75 (0.69-0.79)	0.78 (0.70-0.84)	0.65 (0.57-0.72)	0.77 (0.71-0.82)
Anxietyb symptoms	0.87 (0.85-0.89)	0.89 (0.85-0.92)	0.86 (0.82-0.89)	0.84 (0.79-0.87)	0.85 (0.80-0.88)	0.87 (0.84-0.90)	0.87 (0.83-0.91)	0.87 (0.84-0.90)	0.87 (0.83-0.90)
Internalizingc symptoms	0.86 (0.84-0.89)	0.87 (0.82-0.90)	0.86 (0.82-0.89)	0.84 (0.80-0.88)	0.83 (0.79-0.87)	0.86 (0.83-0.89)	0.86 (0.82-0.90)	0.84 (0.80-0.87)	0.87 (0.83-0.90)
Externalizingd symptoms	0.84 (0.81-0.87)	0.84 (0.80-0.88)	0.84 (0.80-0.87)	0.86 (0.82-0.89)	0.83 (0.78-0.87)	0.83 (0.79-0.87)	0.87 (0.82-0.90)	0.83 (0.79-0.87)	0.81 (0.75-0.85)

Abbreviations: ADHD, attention-deficit/hyperactivity disorder; BT, borderline traits; CD, conduct disorder; CI, confidence interval; DIA-R, Dominic Interactive for Adolescents–Revised; GAD, generalized anxiety disorder; ICC, intraclass correlation coefficient; MDD, major depressive disorder; OCD, obsessive-compulsive disorder; ODD, oppositional defiant disorder; SoPh, social phobia; SPh, specific phobia; SUD, substance use disorder.

^aThe Borderline Traits Scale is defined by the DSM-5–based criteria (n = 9) proposed to establish the borderline personality disorder. This scale is not included in combined internalizing or externalizing symptoms.

^bCombined anxiety symptoms include GAD, SPh, SoPh, and OCD Scales.

^cCombined internalizing symptoms include Anxiety Symptoms, MDD, and Suicidality Scales.

^dCombined externalizing symptoms include ADHD, ODD, and CD Scales.

Criterion-Related Validity

ROC analyses

Table 2 presents results of ROC analyses for 9 DIA-R symptom scales. The AUCs varied from 0.82 to 0.94 for 7 of the 9 scales (GAD, ADHD, ODD, CD, MDD, SoPh, and SUD). It remained moderate for SPh (0.71) but was low for OCD (0.61). Higher cutoff points reaching targeted specificity (≥0.90) were found for all symptom scales, except for SPh (0.84) and OCD (0.83). For these 2 scales, higher specificities were paired with very low sensitivities and deemed statistically unusable. Lower cutoff points reaching targeted sensitivity (≥0.75) were found for all symptom scales, except for CD (0.72), SPh (0.65), and OCD (0.46). The LR+ revealed that adolescents with a given psychopathology were significantly more likely to yield higher or lower cutoff points, except for OCD. The LR+ varied from 2.8 (SPh) to 14.9 (MDD) for higher cutoff points and from 2 (SPh) to 8.7 (SUD) for lower cutoff points.

Table 2.

Results of Receiver Operating Characteristic Analyses for Cutoff Points on DSM-5–Based Symptom Scales of the DIA-R Using the KIDDIE-SADS as an External Validation Criterion (Past 6 Months or Past 12 Months).

Cutoff Points on the DIA-R Scales	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)
MDD, 19 items  Higher cutoff (≥14)  Lower cutoff (≥11)	0.91 (0.86 to 0.96)	0.50 (0.35 to 0.64) 0.79 (0.65 to 0.92)	0.97 (0.95 to 0.98) 0.90 (0.87 to 0.93)	14.9 (8.3 to 32.6) 8.3 (5.9 to 11.8)
GAD, 16 items  Higher cutoff (≥11)  Lower cutoff (≥9)	0.82 (0.74 to 0.90)	0.52 (0.33 to 0.68) 0.77 (0.60 to 0.91)	0.91 (0.88 to 0.93) 0.80 (0.76 to 0.83)	5.7 (3.4 to 9.3) 3.8 (2.8 to 5.1)
SPh, 7 items  Higher cutoff (≥4)  Lower cutoff (≥3)	0.71 (0.62 to 0.80)	0.45 (0.28 to 0.63) 0.65 (0.48 to 0.81)	0.84 (0.80 to 0.87) 0.67 (0.62 to 0.72)	2.8 (1.6 to 4.2) 2.0 (1.4 to 2.6)
SoPh, 8 items  Higher cutoff (≥7)  Lower cutoff (≥5)	0.91 (0.87 to 0.95)	0.56 (0.39 to 0.74) 0.88 (0.75 to 1.00)	0.93 (0.91 to 0.96) 0.84 (0.80 to 0.88)	8.5 (5.2 to 14.4) 5.5 (4.2 to 7.2)
OCD, 6 items  Higher cutoff (≥4)  Lower cutoff (≥3)	0.61 (0.50 to 0.72)	0.29 (0.13 to 0.46) 0.46 (0.29 to 0.66)	0.83 (0.79 to 0.86) 0.65 (0.61 to 0.70)	1.6 (0.7 to 2.7) 1.3 (0.8 to 2.0)
ADHD, 18 items  Higher cutoff (≥14)  Lower cutoff (≥10)	0.85 (0.78 to 0.92)	0.36 (0.10 to 0.64) 0.86 (0.62 to 1.00)	0.92 (0.89 to 0.95) 0.70 (0.65 to 0.74)	4.7 (1.5 to 9.0) 2.8 (2.0 to 3.6)
ODD, 9 items  Higher cutoff (≥7)  Lower cutoff (≥5)	0.86 (0.77 to 0.95)	0.43 (0.23 to 0.63) 0.76 (0.58 to 0.93)	0.97 (0.95 to 0.98) 0.85 (0.81 to 0.88)	13.3 (5.8 to 31.2) 5.0 (3.4 to 7.0)
CD, 15 items  Higher cutoff (≥6)  Lower cutoff (≥3)	0.87 (0.82 to 0.92)	0.38 (0.25 to 0.51) 0.72 (0.59 to 0.85)	0.96 (0.94 to 0.98) 0.84 (0.80 to 0.88)	10.2 (5.5 to 21.3) 4.6 (3.5 to 6.2)
SUD, 6 items  Higher cutoff (≥3)  Lower cutoff (≥2)	0.94 (0.90 to 0.97)	0.66 (0.53 to 0.77) 0.88 (0.77 to 0.96)	0.95 (0.93 to 0.97) 0.90 (0.88 to 0.93)	12.8 (8.2 to 22.8) 8.7 (6.6 to 12.2)

Abbreviations: ADHD, attention-deficit/hyperactivity disorder; AUC, area under the curve; CD, conduct disorder; CI, confidence interval; DIA-R, Dominic Interactive for Adolescents–Revised; GAD, generalized anxiety disorder; KIDDIE-SADS, Schedule for Affective Disorders and Schizophrenia for School-Aged Children; MDD, major depressive disorder; LR+, likelihood ratio positive; OCD, obsessive-compulsive disorder; ODD, oppositional defiant disorder; SoPh, social phobia; SPh, specific phobia; SUD, substance use disorder.

In Table 3, AUCs indicate a high accuracy (>0.90) regarding the ability of the DIA-R BT Scale to predict external criteria. Higher and lower cutoff points with targeted specificity or sensitivity were found. Moreover, adolescents with an Ab-DIB score ≥24 were 10 times more likely to yield the higher cutoff. When suicide plan/attempt were combined with the Ab-DIB as the external criterion, adolescents were 7 times more likely to yield the higher cutoff.

Table 3.

Results of Receiver Operating Characteristic Analyses for Cutoff Points on the DSM-5–Based Borderline Traits Scale of the DIA-R Using the Ab-DIB and Suicidal Ideation with Plan (SIP) or Suicide Attempt (SA) Assessed by the KIDDIE-SADS as External Validation Criteria.

Cutoff Points on the Borderline Traits Scale (9 Criteria)a	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)
Criterion 1: Ab-DIB ≥24 (past 12 months)	0.91 (0.87 to 0.94)
Higher cutoff (≥7)		0.42 (0.28 to 0.58)	0.96 (0.94 to 0.98)	9.6 (5.2 to 18.2)
Lower cutoff (≥5)		0.91 (0.79 to 1.00)	0.81 (0.77 to 0.85)	4.8 (3.9 to 6.2)
Criterion 2: Ab-DIB ≥24 and KIDDIE-SADS (SIP/SA) (past 6 months)	0.91 (0.87 to 0.94)
Higher cutoff (≥7)		0.40 (0.22 to 0.59)	0.94 (0.91 to 0.96)	6.5 (3.1 to 12.4)
Lower cutoff (≥5)		0.96 (0.85 to 1.00)	0.78 (0.74 to 0.82)	4.5 (3.7 to 5.5)

Abbreviations: Ab-DIB, Abbreviated-Diagnostic Interview for Borderlines; AUC, area under the curve; CI, confidence interval; KIDDIE-SADS, Schedule for Affective Disorders and Schizophrenia for School-Aged Children; LR+, likelihood ratio positive.

^aThe Borderline Traits Scale is defined by the DSM-5–based criteria (n = 9) proposed to establish the borderline personality disorder.

When the DIA-R Suicidality Scale was compared with external criteria, it yielded AUCs of 0.81 and 0.85 (Table 4). Higher and lower cutoff points reaching targeted specificity or sensitivity were found. Adolescents with suicidal tendencies were 3 to 9 times more likely to yield cutoff points.

Table 4.

Results of Receiver Operating Characteristic Analyses for Cutoff Points on the Suicidality Scale of the DIA-R Using Suicidal Ideation with Plan (SIP) or Suicide Attempt (SA) Assessed by the KIDDIE-SADS and the BHS as External Validation Criteria.

Cutoff Points on the Suicidality Scale (9 Items)	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)
Criterion 1: KIDDIE-SADS (SIP/SA) (past 6 months)	0.81 (0.74 to 0.89)
Higher cutoff (≥3)		0.58 (0.45 to 0.72)	0.93 (0.90 to 0.96)	8.2 (5.5 to 13.6)
Lower cutoff (≥1)		0.75 (0.63 to 0.87)	0.79 (0.75 to 0.83)	3.5 (2.7 to 4.6)
Criterion 2: KIDDIE-SADS (SIP/SA) and BHS ≥9 (past week)	0.85 (0.76 to 0.95)
Higher cutoff (≥4)		0.58 (0.36 to 0.80)	0.94 (0.92 to 0.96)	9.6 (5.8 to 16.6)
Lower cutoff (≥2)		0.79 (0.60 to 0.95)	0.85 (0.82 to 0.88)	5.2 (3.9 to 7.3)

Abbreviations: AUC, area under the curve; BHS, Beck Hopelessness Scale; CI, confidence interval; KIDDIE-SADS, Schedule for Affective Disorders and Schizophrenia for School-Aged Children; LR+, likelihood ratio positive.

Comparison between clinical and school subsamples of adolescents

Table 5 shows that percentages of mental health problems, detected by the DIA-R, were significantly (P < 0.05) higher in the clinical subsample than in the school subsample, except for SPh and OCD.

Table 5.

Comparison between Adolescents from Clinical and School Subsamples According to the Frequency (%) of the Categories Based on the Cutoff Points of the DIA-R Scales (n = 447).

DIA-R Scales (Cutoff)a	Items/Criteria, n	Clinical Subsample, % (n = 204)	School Subsample, % (n = 243)	χ2 (df = 2)	P Value
MDD	19			51.3	0.000
Likely absent (≤10)		63.2	90.5
Possible (≥11)		16.7	6.6
Likely present (≥14)		20.1	2.9
GAD	16			26.5	0.000
Likely absent (≤8)		59.8	77.8
Possible (≥9)		14.2	14.0
Likely present (≥11)		26.0	8.2
SPh	7			2.0	0.366
Likely absent (≤2)		60.3	60.9
Possible (≥3)		16.7	20.6
Likely present (≥4)		23.0	18.5
SoPh	8			14.1	0.001
Likely absent (≤4)		71.1	82.3
Possible (≥5)		12.3	11.9
Likely present (≥7)		16.7	5.8
OCD	6			0.8	0.661
Likely absent (≤2)		64.2	60.1
Possible (≥3)		18.1	19.8
Likely present (≥4)		17.6	20.2
ADHD	18			23.4	0.000
Likely absent (≤9)		52.0	73.7
Possible (≥10)		32.8	19.8
Likely present (≥14)		15.2	6.6
ODD	9			42.5	0.000
Likely absent (≤4)		59.8	86.4
Possible (≥5)		25.5	10.3
Likely present (≥7)		14.7	3.3
CD	15			55.3	0.000
Likely absent (≤2)		57.8	86.8
Possible (≥3)		26.0	11.9
Likely present (≥6)		16.2	1.2
SUD	6			88.0	0.000
Likely absent (≤1)		58.8	94.2
Possible (≥2)		9.3	4.1
Likely present (≥3)		31.9	1.6
BTb	9			52.0	0.000
Likely absent (≤4)		51.0	82.3
Possible (≥5)		29.9	13.2
Likely present (≥7)		19.1	4.5
Suicidality	9			30.2	0.000
Likely absent (≤0)		61.3	68.7
Possible (≥1)		11.3	22.2
Likely present (≥3)		27.5	9.1

Abbreviations: ADHD, attention-deficit/hyperactivity disorder; BT, borderline traits; CD, conduct disorder; CI, confidence interval; DIA-R, Dominic Interactive for Adolescents–Revised; GAD, generalized anxiety disorder; MDD, major depressive disorder; OCD, obsessive-compulsive disorder; ODD, oppositional defiant disorder; SoPh, social phobia; SPh, specific phobia; SUD, substance use disorder.

^aThe 3-category rating system (likely absent, possible, likely present) is defined by final higher and lower cutoff points.

^bThe Borderline Traits Scale is defined by the DSM-5–based criteria (n = 9) proposed to establish the borderline personality disorder.

Discussion

Reliability

Findings suggest a stability of the reliability coefficients across age, sex, language, and type of subsample (school, clinical). For most of the DIA-R scales, excellent or moderate coefficients^56,69 demonstrated adequate test-retest reliability and internal consistency. Cronbach alpha coefficients observed for SPh and OCD (<0.60) may be explained by the smaller number of questions^43,48 (≤7) and the heterogeneity of phobic situations^43,48 or ruminations/repetitive behaviours. These scales do not assess the DSM-based level of impairment (e.g., distress) required for each symptom.

Criterion-Related Validity

AUCs suggest high (0.91-0.94) or moderate (0.71-0.87) accuracy regarding the ability of the DIA-R current symptom scales to predict mental disorders assessed by the KIDDIE-SADS (past 6 or 12 months). This was not the case for OCD Scale (0.61). Because this scale does not evaluate the DSM-based dimensions required by the KIDDIE-SADS (obsessions/compulsions must be time-consuming and cause significant distress), this may explain its low performance. Moreover, we found that higher and lower cutoff points reached acceptable specificities (≥0.90; 0.84 for SPh) or sensitivities (≥0.72; 0.65 for SPh), except for the OCD Scale. Findings reasonably support the ability of these thresholds to detect tendencies toward mental disorders evaluated by the KIDDIE-SADS.

This study confirms the ability of the DIA-R BT Scale to predict an Ab-DIB score ≥24 (past year) and this score combined with suicide plan/attempt assessed by the KIDDIE-SADS (past 6 months). Results suggest high (>0.90) accuracy estimates defined by AUCs, specificities (higher cutoff), and sensitivities (lower cutoff).

Findings (AUCs >0.80) also support the ability of the DIA-R Suicidality Scale to predict suicide plan/attempt (KIDDIE-SADS) and this criterion combined with a BHS score ≥9 (past week). Higher and lower cutoff points yielding acceptable specificities (≥0.90) or sensitivities (≥0.75) provide different levels of severity regarding suicidal tendencies. However, the DIA-R does not assess all components of explicit intent to die.^18,38

Results from comparisons between clinical and school subsamples on frequencies of categories defined by the final DIA-R thresholds suggest that the higher cutoff may be a more adequate severity index of adolescents’ mental health problems and need for help.^43,48 While the SPh Scale should be used cautiously,^43,48 results obtained for OCD confirm limitations of this scale as a screening measure.

Limitations

Three main limitations should be considered. First, the absence of a “gold standard” for confirming the validity of DSM categories restricts the validity of any instrument assessing these constructs,^43,48,74 including the DIA-R. Second, although the 2 DIA-R thresholds allow for some flexibility, the lower cutoff may overestimate mental disorders in the community.^43,48 Third, as validity estimates (specificity, sensitivity, LR+) depend on the distribution of traits/disorders in the population,^70,73,75 results must be interpreted in the context of the population sampled.

Clinical Utility of the DIA-R

In an “evidence-based assessment” approach,^76,77 the convergence between the DIA-R with clinically relevant measures increases the potential utility of this instrument for clinical practice.

Early identification of psychopathology represents a major issue.^{11,14,31,37,77–79} The DIA-R could be useful in frontline services to assess several DSM-5–based mental health problems, suicidal tendencies, and comorbidity within a brief administration time. Categorical and dimensional data, which are available immediately after the self-administration, could be used in conjunction with results from other measures to identify high-risk adolescents.

The DIA-R could also be useful as a screening instrument before the psychiatric interview takes place or in the context of a clinical follow-up to evaluate the treatment response.⁴⁸ Categorical data may provide relevant information on the adolescent’s perception of symptoms/criteria defining disorders, borderline traits, and suicidality in the context of a multi-informant and multimethod approach. However, clinical judgement (best estimate) is always necessary in decision making about definite diagnosis and intervention.^{40,43,44,48,80}

Conclusions

Our findings clearly highlight that the DIA-R, with its helpful multimedia self-report format and its additional scales, is a suitable and relevant instrument for assessing a wide range of adolescents’ mental health problems in the continuum of services and in research. However, OCD is an exception to this conclusion. Since its internal consistency and validity were very low, OCD should not be retained in the final version of the DIA-R.

Future research should examine the reliability and the validity of the DIA-R in specific high-risk subgroups of adolescents, notably those with learning disabilities or language impairment.^81,82 The DIA-R developmentally sensitive approach⁴⁹ could be helpful for assessing these youngsters. Research about methods for combining data generated by the DIA-R with data from other standardized measures to determine the presence of a mental disorder remains a major clinical issue.⁴⁸ Finally, the longitudinal predictive validity of the DIA-R represents a fundamental step for future research.⁴³