Assessing the diagnostic validity of a structured psychiatric interview in a first-admission hospital sample

Abstract

The use of structured psychiatric interviews performed by non-clinicians is frequent for research purposes and is becoming increasingly common in clini-cal practice. The validity of such interviews has rarely been evaluated empirically. In this study of a sample of 100 diagnostically heterogeneous, first-admitted inpatients, the results of an assessment with the Structured Clinical Interview for DSM-IV (SCID), yielding a DSM-IV diagnosis and performed by a trained non-clinician, were compared with a consensus lifetime best diagnostic estimate (DSM-IV) by two experienced research clinicians, based on multiple sources of information, which included videotaped comprehensive semi-structured narrative interviews. The overall kappa agreement was 0.18. The sensitivity and specificity for the diagnosis of schizophrenia by SCID were 19% and 100%, respectively. It is concluded that structured interviews performed by non-clinicians are not recommendable for clinical work and should only be used in research with certain precautions. It is suggested that a revival of systematic theoretical and practical training in psychopathology is an obvious way forward in order to improve the validity and therapeutic utility of psychiatric diagnosis.

Structured psychiatric interviews are now the diagnostic gold standard in psychiatric research and are making a rapid inroad into daily clinical work. In research, non-clinicians equipped with structured interviews often perform diagnostic assessments. Structured interviews have been shown to yield high diagnostic reliability among novice interviewers 1.

A structured interview is defined as “an interview consisting of… predetermined questions presented in a definite order”. These questions “yield diagnostic information based on the patient’s responses and the interviewer’s observations. The interviews… identify symptoms and syndromes which meet specific diagnostic criteria” 2.

Structured interviews made their appearance as part of the operational revolution in psychiatry, in the quest of improving diagnostic reliability. They were strongly advocated for by a major figure of the DSM-III project, Robert Spitzer, in a seminal article entitled “Are clinicians still necessary?” 3. The potential unreliability in the quality and quantity of the diagnostic information elicited across the patients (“information variance”) is here countered by the application of identical questions, presented to the patients in a fixed sequence. Another source of unreliability, potentially involved in the process of converting clinical information into diagnostic criteria (“criterion variance”), is minimized by formulating the interview questions in a wording as close as possible to the phrasings of the diagnostic criteria. In sum, the structured interview reduces the initiative, inference and reflection by the interviewer almost to zero, obviating clinical psychiatric experience and education in psychopathology, thus allowing a suitably trained non-clinician to perform the diagnostic assessment.

The validity of structured interviews has rarely been explored 4,5. Moreover, it is sometimes claimed to be untestable in principle, due to the unavailability of a “gold standard” 3,6, a claim implicitly reiterating the premise that clinicians’ assessments cannot serve this purpose because of their unreliability 6,7. This argument is non-sequitur, however. Although clinicians may vary in their knowledge and skills, the fact is that our current classification is predominantly anchored in descriptive, phenomenological distinctions. To the best of our knowledge, no psychopathological-conceptual or phenomenological argument in favor of structured interviews has ever been advanced (apart from pointing to the unreliability of clinicians).

In this study, which is part of a larger project on the conceptual and empirical foundations of psychopathological assessment, we examined the diagnostic validity of a structured interview, performed by a suitably trained non-psychiatrist, in a diagnostically heterogeneous, first-admission hospital sample.

Our measure of validity was the consensus lifetime best estimate diagnosis 8, based on a semi-structured conversational interview conducted by a reliability-trained, experienced psychiatrist, followed by an independent review of the diagnostic material by another senior clinician and a final consensus evaluation using all existing sources of diagnostic information.

We chose the Structured Clinical Interview for DSM-IV (SCID, 9) as a representative structured interview because it is very frequently used (e.g., a PubMed search conducted on November 16, 2011 showed that 11 out of the 15 latest publications on schizophrenia in which a diagnostic evaluation was carried out used the SCID). The SCID is designed to be an efficient, user-friendly, and reliable clinical interview for making DSM diagnoses 10. It incorporates the use of obligatory questions, corresponding to DSM-IV operational criteria, and an algorithm for arriving at the final diagnoses. The pre-formulated questions can be answered with a ”yes” or ”no”. It is possible to ask for more detailed descriptions. Yet, the SCID user’s guide stresses the importance of asking the questions as formulated in the interview schedule: “Do stick to the initial questions as they are written…” 11. The SCID has been found to yield highly reliable diagnoses for most axis I disorders 1,5,12,13.

METHODS

Sample

The study was carried out at the Psychiatric Center Hvidovre, a department of the University of Copenhagen providing psychiatric service to a population of 150,000 in a catchment area of the City of Copenhagen (there are no private psychiatric in-patient facilities in Denmark). The department has a rich and long psychopathological research tradition of adoption, high risk, linkage and clinical sample studies in schizophrenia, most recently in the domain of anomalies of self-experience 14,15,16,17,18.

All consecutive first admissions to the department were screened for eligibility over 18 months, starting from June 2009, independently of their clinical diagnosis. In order to be included, patients had to be considered capable of tolerating lengthy interviews (which naturally excluded aggressive, agitated or severely psychotic patients) and to provide informed consent. Exclusion criteria comprised primary or clinically dominating alcohol/substance abuse, history of brain injury, mental retardation, organic brain disorder, and age >65 years. Due to ethical concerns, involuntarily admitted and legal patients (both categories representing an important proportion of first-admitted inpatients) were also excluded.

Six patients had to be excluded after enrolment because, upon closer examination, they did not meet the inclusion criteria (n=3), did not show up for the interview appointments (n=2) or withdrew consent after completed interviews (n=1). Sixteen patients declined to participate in the study (clinical diagnoses: 4 with schizophrenia, 1 with schizotypal disorder, 9 with major depression, 1 with anxiety and 1 with deferred diagnosis).

The final sample consisted of 34 men and 66 women (sex distribution reflecting the selection process), with a mean age of 27.7 years (range 18-65 years), representing 82% of the patients initially invited to participate.

Interviews

All patients were interviewed twice during the same week. The mean time from admission to the first interview was 13 days (range 2-71). All interviews were videotaped.

The first interview was conducted by a recent MA in clinical psychology (RR), specifically trained and certified as a SCID-interviewer by the University of California Los Angeles Center for Neurocognition and Emotion in Schizophrenia. The interview consisted of the SCID-I and the Schizotypal Personality Disorder module from the SCID-II. The interview was conducted in a fully structured way: the interviewer faithfully asked the pre-formulated questions in the predefined order. The interviewer was allowed to ask for more information, and to modify the rating if relevant new information emerged during the interview. The average length of the interview was 1.5 hours. An experienced research psychiatrist supervised the performance of the interview and the allocation of the DSM-IV diagnosis, in order to prevent errors.

The second interview was carried out by an experienced psychiatrist (JN). This was a semi-structured conversational interview (SSCI), including a thorough psychosocial history, a description of the illness evolution (including the circumstances leading to admission), the Operational Criteria Checklist (OPCRIT, 19) expanded with additional items from the Schedule for Affective Disorders and Schizophrenia (SADS-L, 20), the Examination of Anomalous Self-Experiences scale (EASE, 21), the perceptual section from the Bonn Scale for the Assessment of Basic Symptoms 22, and an assessment of first rank symptom continua as described by Koehler 23 and of abnormal expressive features (e.g., affect modulation, contact quality, gaze, stereotypies, mannerisms, disorganization, and disorder of language) 15,16,24.

The interviewer explored the items in a sequence that was felt appropriate and adequate to the subject’s own concerns and responses, according to the phenomenological principles proposed by Jaspers and others 25. The structure of the interview relied on the interviewer’s obligation to score all items. Yet, the concrete, practical conduct and sequence of the interview was dictated by the dynamics and context of the encounter; i.e. the style was free and conversational. The questions were contextually adapted and followed the logic of the patient’s narrative, typically asking for more details or further examples. The patient was encouraged to speak freely, was rarely interrupted, and was given time for reflection and recollection. Scoring of a symptom was never based on a simple yes/no answer, but always required self-descriptions, i.e. the examples formulated in the patient’s own words. The average time to complete the SSCI was 3.5 hours, sometimes requiring a splitting of the interview into two separate sessions. On the basis of the interview, JN allocated the DSM-IV diagnosis.

The project director (JP), a senior research clinician, independently reviewed the diagnostic material elicited in the SSCI and allocated his own DSM-IV diagnosis. There was a diagnostic agreement between JN and JP in 93% of patients.

Finally, the consensus lifetime best estimate (CLBE) of DSM-IV diagnosis was allocated to each patient at a consensus meeting of JP and JN, using all information available (videos, notes, charts’ clinical information, including second informant descriptions) on each patient. This DSM-IV diagnosis was the study’s gold standard.

Statistics

Kappa statistics was used to calculate the agreement between the diagnostic procedures. The sensitivity of the SCID was calculated as the number of true positives divided by the sum of true positives and false negatives. The specificity was calculated as the number of true negatives divided by the sum of true negatives and false positives.

For the purpose of the analyses, we operated with the following hierarchy of DSM-IV diagnostic categories: 1. schizophrenia, 2. other (non-affective) psychosis, 3. bipolar illness, 4. major depression, 5. schizotypal disorder, 6. other diagnosis. The category “other diagnosis” contains mainly anxiety disorders, obsessive-compulsive disorder and personality disorders other than schizotypal.

Results

The cross-tabulation of SCID and CLBE diagnoses appears in Table 1. The kappa of the overall diagnostic concordance between these two approaches was 0.18. The corresponding kappa between SSCI and CLBE was 0.92. The kappa agreement between the SCID and the CLBE with the sample dichotomized into the schizophrenia spectrum (schizophrenia, other psychosis, schizotypal disorder) vs. not-spectrum (all other diagnoses combined) was 0.31.

Table 1.

Table 1 DSM-IV diagnoses made by a trained non-clinician using the Structured Clinical Interview for DSM-IV (SCID) vs. those made by two experienced clinicians on the basis of a consensus lifetime best estimate

		Best consensus diagnoses
		Schizophrenia	Non-affective psychosis	Schizotypal disorder	Major depression	Bipolar	Other	Total
SCID	Schizophrenia	8	0	0	0	0	0	8
d	Non-affective psychosis	8	0	1	0	0	1	10
i	Schizotypal disorder	8	0	5	0	0	0	13
a	Major depression	10	2	11	14	0	13	50
g	Bipolar	1	1	2	0	0	0	4
n	Other	7	1	2	0	1	4	15
o	Total	42	4	21	14	1	18	100
s
e
s

Using the CLBE as gold standard, the sensitivity and specificity of the SCID for schizophrenia alone were 19% and 100%, respectively. The corresponding figures for all non-affective psychoses combined (i.e., schizophrenia and other non-affective psychosis) raised to 34% and 96%. Finally, the sensitivity and specificity of the SCID for the schizophrenia spectrum (schizophrenia, other non-affective psychosis and schizotypy) were 44% and 97% respectively. If the hierarchy was altered to let schizotypal disorder precede affective disorders, the sensitivity of SCID for the schizophrenia spectrum raised slightly to 53%, whereas the specificity remained unchanged (97%).

Discussion

A limitation of this study is that the patient selection tended to eliminate flamboyant psychotic cases, making the sample diagnostically more “difficult”, thus perhaps amplifying the shortcomings of the structured interview.

The overall agreement between the SCID-derived and CLBE DSM-IV diagnoses was very low (kappa 0.18). The SCID tended to diagnose more patients with major depression and fewer with schizophrenia and schizotypal disorder. It was clear that the major source of the diagnostic differences was information variance. A brief, or sometimes monosyllabic response to a question of the structured interview was often short of crucial psychopathological information that might have been elicited in an epistemologically more adequate manner.

Fennig et al 4 showed a high diagnostic concordance for schizophrenia and bipolar disorder between the SCID diagnosis by non-clinician interviewers and the best consensus diagnostic estimates performed by psychiatrists. However, that particular study was limited to patients who were already diagnosed with a psychotic condition. More importantly, the SCID-elicited information also served as the main source of the best consensus estimate. This implies little information variance between psychiatrists and non-psychiatrists, i.e. clinician- and non-clinician raters made their diagnoses upon similar information. Minimizing information variance is exactly one of the cornerstones of the structured interview, but it does not guarantee the quality (validity) of the information.

As already mentioned, the operational revolution in psychiatry was motivated by a lack of diagnostic reliability, incompatible with psychiatry’s scientific aspirations. The operational criteria and the corresponding structured interviews emerged as a solution. The structured interview aspires to achieving a quasi-experimental, stimulus-response purity of the behavioristic paradigm 26,27, in order to circumvent or shortcut the complexities of human subjectivity, discourse and communication, which are always at play in patient-clinician psychiatric exchange 28,29. The crucial issue of how to elicit, explore and convert the patient’s experience (first-person perspective) into a third-person data format, used for the diagnostic allocation, is unaddressed. On the contrary, a basic assumption is the confidence in the face value of “yes/no” answers, for instance, that a “yes” is confirmatory of the diagnostic criterion being asked about. That confidence reflects another implicit belief, namely that the symptoms have their mode of existence as ready-made, well-defined and well-articulated mental objects in the patient’s consciousness, only waiting for adequate prompting in order to appear in full view.

This study only examines the validity of a structured interview performed, lege artis, by a specifically trained non-psychiatrist. We have no data on the potential validity of a structured interview schedule in the hands of an experienced, reliability-trained clinician. Such a clinician, taking advantage of the possibility of asking for more detailed information (allowed for in the interview guide), may likely arrive at valid diagnoses, but that would happen in a manner not different in kind from the semi-structured interview employed in this study. In other words, once the clinician-interviewer is granted the liberty for contextually appropriate (i.e., semi-structured) in-depth conversation with the patient, any particular interview scheme would probably do equally well (provided that the clinician is knowledgeable, skilled and reliability-trained).

An unintended consequence of the operational revolution has been a general decline in psychopathological competence 30 and no strikingly evident improvement in the reliability and the validity of clinicians’ diagnoses. The way forward, both in research and clinical work, is to revive teaching and training of psychopathology 31. Unfortunately, such education today is often reduced to a training in the use of a particular, locally selected, interview schedule. Yet, it is not enough to memorize the questions of a scheme and the diagnostic criteria. Teaching psychopathology implies targeted literature studies, weekly peer discussions of concepts (e.g., what does the concept of delusion or hallucination signify; are their operational definitions conceptually coherent and phenomenologically faithful?), and live, supervised interviews, followed by discussions of the interview technical aspects, interpersonal interaction, and the nature and diagnostic significance of the elicited information.

In conclusion, this study demonstrated a poor diagnostic performance of a structured psychiatric interview conducted by a for-the-purpose trained non-clinician. Such an interview approach cannot be recommended for clinical work, due to a high proportion of misdiagnoses. The high specificity for the diagnosis of schizophrenia (100%) may suggest a limited applicability of such interviews in research with the purpose of a confirmation of an antecedent, clinician’s diagnosis.