The Validity of Psychiatric Diagnosis Revisited

Abstract

Background: The authors reviewed the types and phases of validity of psychiatric diagnosis. In 1970, Robins and Guze proposed five phases to achieve valid classification of mental disorders: clinical description, laboratory study, exclusion of other disorders, follow-up study, and family study. Objectives: The objectives of this paper are to review what has been learned since Robins and Guze's influential article as well as examine the impact of the new discoveries in neurosciences and neuroimaging on the practicing clinician. Method: The authors reviewed the literature on the concept of validity in psychiatry with emphasis on the role of clinical training, the use of structured interviews and rating scales, and the importance of the new discoveries in neurosciences. Results: Robins and Guze's phases have been the cornerstone of construct validity in psychiatry at the level of researchers. In the absence of the gold standard of psychiatric diagnosis, Spitzer proposed the “LEAD,” which is an acronym for longitudinal evaluation, and is done by expert clinicians utilizing all the data available. The LEAD standard is construct validity at the level of experts; however, guidelines are lacking to improve the validity skills of the practicing clinicians. Conclusions: The authors propose the acronym DR.SEE, which stands for data, reference definitions, rating scales, clinical experience, and external validators. The authors recommend that clinicians use the DR.SEE paradigm to improve the validity of psychiatric diagnoses.

Introduction

Validity and reliability are two important topics vital to the development of modern psychiatry. Reliability refers to the extent to which an experiment, test or any measuring procedure yields the same results on repeated trials,¹ and is the topic of another paper. Validity is a more difficult term to define because its meaning differs based on the context. Validity, in a very general sense, refers to examining the approximate truth or falsity of scientific propositions.² When applied to measuring instruments, validity refers to how well the instrument measures what it purports to measure.¹ When applied to a disease entity, such as bacterial pneumonia, validity refers to the evidence that bacteria is the cause (verified by sputum culture), lung pathology exists (confirmed by x-ray findings), the symptoms (shortness of breath, fever, and cough), and signs (tachpnea, rales) are compatible with etiology and the disease responds to appropriate antimicrobial treatment. In a psychiatric illness, the patient comes with a subjective complaint (e.g., anxiety, depression, paranoia), and the trained clinician elicits signs of the illness through observation of the patient's demeanor, behavior, and thought process. However, there are fewer definitive objective measures (akin to x-ray and sputum culture) that confirm the diagnosis

During the first half of the 20th century, psychiatrists and other mental health clinicians were not particularly interested in making diagnoses, mainly due to an emphasis on a psychoanalytic approach. Beginning in the 1950s, clinicians began to label psychiatric disorders as particular diagnostic entities. At about the same time, psychiatry began to adopt the medical model. This model assumes that a disease, a syndrome, or a disorder has three components: an etiological agent, a pathological process, and symptoms and signs. The etiology, pathology, and even the treatment of any disease, syndrome, or disorder may be known or unknown.³

In order to adopt the medical model, the field of psychiatry needed a new and comprehensive classification system. The development of a classification system of mental diseases has been a major effort of the World Health Organization (WHO) from its publication of the sixth revision of the International Classification of Diseases (ICD-6) in 1948 until the present time.⁴ The World Health Organization (WHO) has also published several manuals on the diagnostic criteria of mental disorders and the International Classification of Diseases (ICD); the 10th edition, published in 1993, is the latest.⁵ On this continent, the American Psychiatric Association Committee on Nomenclature and Statistics developed and published in 1952 the first edition of the Diagnostic and Statistical Manual: Mental Disorders (DSM-I).⁶ Several publications followed, and the latest is the fourth edition of the DSM (DSM-IV) published in 1994, which includes the diagnostic criteria of all psychiatric disorders.⁷ Most of the psychiatric abnormalities are called disorders because the etiology is unknown. Nevertheless, effective treatments have been developed and utilized for mental disorders without the etiology or pathology being fully elicited.³

The goals of this paper are to learn about the new concepts on the validity of psychiatric diagnosis and the impact of the new discoveries in neurosciences on practicing clinicians, such as psychiatrists, clinical psychologists, and therapists.

Methods

Computerized literature searches were conducted using MEDLINE and PsychInfo. Searches were conducted using entries from January, 1970, to December, 2004, that were published in English. Searches from Medline were conducted for entries that contained the words “Validity” and “Psychiatric Diagnosis.” This result yielded 85 citations. Similar searches from PsychInfo using the same search criteria yielded 145 citations. Searches from Medline were conducted for entries that contained the words “Validity” and “Structured Interviews.” This result yielded 249 citations. Similar searches from PsychInfo using the same search criteria yielded 545 citations. Additionally, relevant references attached to published papers were also reviewed while the authors identified more papers and books through consultations with colleagues and experts in the field. The authors were looking for new knowledge on the concept of validity of psychiatric diagnosis and how clinicians utilize validity skills in routine clinical practice

Results

Types of validity. There are four main types of validity: content, criterion, construct and procedural.

Content validity. Content validity refers to the degree to which an empirical measurement reflects a specific domain of content. An arithmetical operations test is content valid if it includes addition, subtraction, multiplication, and division.¹ In medicine and psychiatry, clinicians agree on important features that make up a disease, a syndrome, or a disorder. Neurologists agree that Parkinson's disease has three main features: slow movement (bradykinesia), increased tone, and resting tremor. Psychiatrists agree that a patient with schizophrenia has delusions, hallucinations, disorganization, and bizarre behavior. Typically, the items that represent the domain or disorder are derived from the consensus of experts in the field. Content validity facilitates communication among clinicians and provides an initial framework for further validation.^3,8 The worldwide use of the DSM and ICD diagnostic criteria reflects great progress with regard to content validity because clinicians across the globe use the same nomenclature of mental disorders and know the specific criteria of each proposed disorder.⁹

Criterion validity. Criterion validity is measuring something that is external to the measuring instrument itself, called the criterion.¹⁰ Internists agree that diabetes mellitus has four main symptoms: polyuria, polyphagia, polydipsia, and unexplained weight loss (content validity). In criterion validity, an external measure is used to validate the diagnosis that is made by content validity. For example in diabetes, findings would include an abnormal glucose tolerance test or fasting blood sugar. Depending on the timing of the measurements, the criterion validity can be postdictive, concurrent, or predictive. Postdictive validity entails correlating the criterion that happened in the past with the present (e.g., elementary school performance and high school grades). Concurrent validity correlates the measuring instrument with some criterion measured at the same time (e.g., x-ray finding of a broken humerus in a swollen painful arm). Predictive validity correlates a measuring instrument with a criterion that will be assessed in the future (e.g., college admission test scores and graduation four years later)

A biological marker was defined by Buchsbaum as a measurable indicator of a disease, which may or may not be causal,¹¹ and is a good example of criterion validity. Several biological markers have been studied in psychiatry, such as platelet monoamine oxidase (MAO), dexamethasone suppression test, metabolites of serotonin and noradrenaline in the cerebrospinal fluid, and others. Decades of research on biological markers have resulted in some promising results. However, no single biological marker has been unequivocally identified as a marker for mental disorders.^11–14 In light of the absence of biological markers for mental disorders, Spitzer proposed the LEAD standard.¹⁵ LEAD is an acronym for longitudinal evaluation, and is done by expert clinicians who utilize all the data available. The LEAD standard is an important step toward obtaining the best estimate diagnosis by requiring expert clinicians to utilize all the available data over time, including information from family members, hospital records, psychological evaluation, and laboratory results. The requirement of LEAD to have expert clinicians make independent assessments, discuss diagnostic disagreement, and make a consensus diagnosis accounts for the difficulty in implementing the LEAD standard and its limited use.^15–17

Construct validity. Construct validity refers to the extent to which a particular measure relates to other measures consistent with theoretically derived hypotheses.¹ Typically, researchers formulate a hypothesis (construct) that a variety of behaviors will correlate with one another. For example, the construct of diagnosis of schizophrenia relies on the young age onset, the presence of psychosis, the absence of organic cause of psychosis, and positive family history of schizophrenia. The construct of dementia relies on later onset of the illness, impairment of short- and long-term memory, disturbances of higher cortical function (e.g., aphasia), and psychological testing consistent with dementia. Construct validity is woven into the theoretical fabric of social sciences and psychiatry because of the absence of criterion validity.^1,18 Construct validity boils down to the circumstantial evidence for the usefulness of the construct or the hypothesis under study.¹⁰

In 1970, Robins and Guze proposed five phases to achieve valid classification of mental disorders: clinical description, laboratory study, exclusion of other disorders, follow-up study, and family study.¹⁹ Robins and Guze actually were the first to articulate the elements of construct validity in psychiatry. They applied the criteria to schizophrenia and concluded that good prognosis schizophrenia is not a mild schizophrenia but a different illness. The point of Robins and Guze's phases was to redefine psychiatric disorders over time so that the diagnostic criteria more and more closely approximate the true definition of the disorder, the ultimate goal of validity. Other authors have added more potential validators, such as treatment response and diagnostic consistency over time.^20–22 It is very important to note that construct validity is the product of clinical experience, clinical research, laboratory, epidemiological, and other research data. Construct validity requires a pattern of consistent findings across studies involving different samples and different settings.¹⁸

Procedural validity. Procedural validity refers to the adequacy of a new diagnostic procedure in replacing or simulating some existing procedure.³ For example, one may use a structured interview to replace the existing procedure of an open-ended interview by a clinician. Because of the widespread use of DSM and ICD, many efforts were directed toward finding different procedures that approximate the “ideal” application of DSM and ICD criteria. Green and Price developed a short form of the Schedule for Affective Disorders and Schizophrenia (SADS) to encourage psychiatrists to be involved in clinical research.²³ It is very important to remember, though, what Spitzer said regarding these efforts: “Procedural validity speaks only to the issue of the validity of the evaluation procedure and not to the validity of the diagnostic categories themselves.” The validity of the diagnostic categories of the DSM and ICD is extensive and beyond the scope of this paper.^24–29

Phases of Validity

Ideally, the validity of psychiatric diagnosis has three phases.

Phase I. The patient has specific complaint(s) addressed to the clinician. The clinician needs to determine whether the patient has real symptoms or the patient is feigning symptoms for secondary gain. The clinician needs to measure the symptoms, observe the patient's behavior, and make a provisional diagnosis. Content validity plays an important role in this phase

Phase II. 1) The clinician collects more data (e.g., from family, old records) and orders laboratory, psychological, or imaging studies as indicated. 2) The clinician formulates an entity with suffix disease, syndrome, or disorder. 3) The clinician initiates treatment to alleviate the suffering of the patient.

Typically, phases I and II happen during the first visit of the patient because the patient needs immediate treatment and cannot wait for full validation. Moreover, it is important to remember that the clinician's goal is not quest of knowledge per se, but the ability to use the available knowledge and skills to prevent and diminish the suffering and disabilities of the patient.²⁰ Content and construct validity play important roles in phases I and II.

Phase III. The clinician collects more evidence that may confirm or refute his initial diagnosis. The course and the progression of the illness and the response to treatment can provide more valuable information to the clinician. The new evidence collected in this phase can result in redefining or changing the diagnosis. Construct validity continues to play a major role in this phase. Validity is an ongoing process and may continue beyond the termination of a particular doctor-patient relationship.

Discussion

Validity criteria and gold standard in diagnosis: New definitions Validity criterion was defined by Aboraya as any knowledge, method (e.g., rating scale or structured interview), or procedure (e.g., blood test, lumbar puncture, or MRI) that can improve the accuracy of the disease, syndrome, or disorder measurement, help to rule out other diseases, syndromes, or disorders in the differential diagnosis, or validate a provisional diagnosis of the disease, syndrome, or disorder.⁹ Validity is a relative phenomenon and any knowledge provided by the validity criteria helps researchers and clinicians to validate the construct of the disease, syndrome, or disorder. As Nunnally has said, “Validity usually is a matter of degree rather than an all-or-none property, and validation is an unending process.”¹⁰

In psychiatry, the lack of biological markers has led many investigators to conclude that psychiatry lacks a “gold standard.”^30,31 We define the gold standard in diagnosis as the standard that utilizes all the validity criteria available at the time. In medicine and psychiatry, clinicians should use all the available validity criteria to obtain the most accurate diagnosis. The more validity criteria used, the more accurate the diagnosis. The psychiatrist who uses his or her clinical skills along with a structured interview can provide a more accurate diagnosis of schizophrenia in comparison with the psychiatrist who uses clinical skills alone. Similarly, the neurologist who uses clinical examination, lumbar puncture, and MRI can provide a more accurate diagnosis of multiple sclerosis in comparison with the neurologist who uses clinical examination alone.

Proposal to improve the validity skills of clinicians. The literature on the concept of validity lacks guidelines that can improve the validity skills of practicing clinicians. Aboraya and Compton proposed the acronym DR.SEEK, stands for data, reference definitions, standardized instruments, clinical experience, external validators, and knowledge to improve the accuracy of making psychiatric diagnoses.⁹ In this paper, we propose the DR.SEE paradigm, which is the acronym for data and knowledge, reference definitions, rating scales, clinical experience, and external validators. The DR.SEE paradigm is a clinician's form of the original DR.SEEK and LEAD paradigms with a focus on the day-to-day practice of clinicians. By using the DR.SEE paradigm, clinicians can improve procedural and construct validity.

Data and knowledge. In comparison to medicine, psychiatric information goes well beyond the individual patient. A patient who has paranoid delusions and shoots a shotgun at the neighbor may deny having any paranoid thoughts. Clinicians use their skills to build a rapport with patient, use the appropriate proxy information sources and observe the patient's behavior to get the most valid data. Clinicians should obtain and utilize all data essential for an accurate diagnosis: the patient's clinical picture, history and course of illness, family information, family history, psychological testing or any other pertinent data in the particular case. The use of all the available data can improve the diagnostic validity.^15,31,32 The knowledge and education of mental health clinicians is key to adequately make a diagnosis through this clinical assessment. A minimum master's degree in a mental health field, such as psychology, with a clinical emphasis or a medical degree with psychiatry residency training is recommended to give the clinician adequate knowledge to make psychiatric diagnosis.

Reference guide. Reference definition refers to the definition of psychiatric symptoms and their levels of severity. First, clinicians need to define and agree among themselves on the definitions of psychiatric symptoms. Second, when clinicians ask patients questions, they should convey the meaning of the questions to their patients. Additionally, clinicians should understand the expression of symptoms in different cultures. Psychiatric symptoms reported by the patient, assessed and observed by the clinician are the main source of information the clinician utilizes to diagnose and treat the patient.³³ In other words, the measurement of psychiatric symptoms is still the main source to assess whether the diagnostic criteria of the disorder are met. Structured interviews and rating scales can help to define the meaning of terminology and differentiate the levels of severity of symptoms.^34,35

Rating scales and structured interviews. The use of standardized or semistandardized instruments helps the clinician in many ways. First, standardization forces the clinician to cover all the areas of psychopathology under question. Second, standardization provides similarities in the way questions are asked and minimizes variability among clinicians. Standardization applies to the detailed structured or semistructured interviews, such as the Schedules for Clinical Assessment in Neuropsychiatry (SCAN), Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I), Diagnostic Interview for Genetic Studies (DIGS), Mini-International Neuropsychiatric Interview (MINI), and to the rating scales such as Positive and Negative Syndrome Scale (PANSS), Brief Psychiatric Rating Scale (BPRS), Hamilton Rating Scale for Depression (Ham-D), and others.^34–40

Although the use of structured interviews can provide a more accurate diagnosis in comparison with routine clinical diagnosis, most clinicians do not use them for three main reasons. First, structured interviews are time-consuming; a SCAN or SCID interview lasts from 1 to 2 hours. Second, structured interviews are cumbersome, complicated, and interfere with establishing a rapport with the patient. Third, many structured interviews require lengthy and extensive training.

On the other hand, rating scales take less time and can help clinicians to obtain more accurate data. Many rating scales with good reliability are available.⁴¹ Examples of these include the following: Hamilton Rating Scale for Depression (Ham-D), Abnormal Involuntary Movement Scale (AIMS), Conner's Rating Scale-Revised (CRS-R), Instrumental Activities of Daily Living Scale (IADL), Mini Mental State Examination (MMSE), Young Mania Rating Scale (YMRS), Yale-Brown Obsessive Compulsive Scale (Y-BOCS), and others.^36,42–47 These rating scales can be especially useful when information needs to be obtained from others who observe the patient's behavior (e.g., parents and teachers for the Connors, caretakers for the IADL) or to quantify observable signs of illness and measure changes with treatment (e.g., level of irritability for the YMRS or degree of depressed mood for the HAM-D). These also provide benchmarks for comparing a particular patient with those who have been studied in treatment trials.

Experience. Mental health clinicians may be psychiatrists, clinical psychologists, therapists, or others who have actual experience and contact with patients with mental disorders. Clinical experience with psychiatric abnormalities and the development of skills to elicit them and ascertain their significance have been viewed as the reference standard of psychopathology assessment.⁴⁸ As reading textbooks of medicine alone does not qualify the reader to diagnose medical disorders, reading textbooks of psychiatry does not qualify the reader to diagnose mental disorders. Clinical experience is indispensable when it comes to diagnoses of psychotic, bipolar, and personality disorders, especially with regard to judging the significance of symptoms.^49,50 This clinical experience cannot be replaced with a few weeks of training, nor is it exclusive to psychiatrists. Several years of experience with inpatient and outpatient populations are required to gain the appropriate experience to adequately diagnose psychiatric disorders. The longer the experience of the clinician, the more likely the diagnosis is accurate

External Validators. A biological marker was defined earlier as a measurable indicator of a disease, which may or may not be causal. External validators, on the other hand, are elements external to the disease definitions and are not restricted to biological markers. The past decade has witnessed an explosion in brain imaging techniques allowing scientists to study brain structures and function even at a cellular and molecular level. Structural magnetic resonance imaging (MRI), functional MRI (fMRI), magnetic resonance spectroscopy (MRS), single proton emission computed tomography (SPECT), and positron emission tomography (PET) are some of these new brain-imaging techniques. Andreasen has used the term new external validators for these brain imaging techniques and other new branches of neuroscience and has emphasized their importance in understanding the relationship between individual symptoms and the changes in structure and/or function of the brain.⁵¹ Although these new techniques have yielded important research findings, these findings cannot yet generally be translated into clinical practice.⁵² However, one area where neuroimaging is actually emerging as a diagnostic external validator is in the use of PET scans to detect early Alzheimer's disease.⁵³ The new external validators hold promise of validating psychiatric diagnosis and predicting treatment response in psychiatry in the near future.⁵⁴

Although the current available techniques are not useful as indicators of the presence of psychiatric disorders, they are useful to rule out other disorders in the differential diagnosis. For example, clinicians use the computed tomography (CT) and/or MRI to rule out trauma, stroke, or multiple sclerosis as the cause of depressive or psychotic symptoms. As another example, hormone levels (e.g., thyroid hormones) are measured to exclude hypothyroidism or hyperthyroidism as the cause of anxiety or depression

The application of DR.SEE paradigm. The following three cases show that the use of DR.SEE paradigm helps in making a valid diagnosis

Case one. The patient is a 33-year-old female with multiple psychiatric hospitalizations since the age of 19. The patient's main psychiatric symptoms are paranoid delusions, auditory hallucinations, and manic symptoms (e.g., pressured speech, grandiosity, and racing thoughts). The patient had several psychiatric diagnoses, including paranoid schizophrenia, schizoaffective disorder, bipolar type, and bipolar disorder with psychotic features. During the last admission, her primary symptoms were paranoid and grandiose delusions and auditory hallucinations. The clinician gathered data by interviewing the patient and the family and reviewing the old records. This investigation indicated that the patient had hypomanic symptoms, such as pressured speech, grandiosity, and racing thoughts, which lasted a very short time compared to the duration of delusions and hallucinations. The clinician interviewed the patient using the Schedules for Clinical Assessment in Neuropsychiatry (SCAN), and the patient had Schneider's first-rank symptoms, such as voices arguing and voices commenting on the patient's actions. The clinician used the reference definitions of the SCAN glossary. Organic work up for the patient was done and the results were normal. Using the DR.SEE paradigm, the final diagnosis was schizophrenia, paranoid. The patient was treated successfully with antipsychotic without any mood stabilizers

Case two. Mr. P was a 58-year-old man referred to the neurology clinic by his primary care physician in July of 2004 for memory difficulties. Initial workup was begun, which included neuropsychological testing in August of 2004. The patient was hospitalized on the neurology service in February, 2005, when he presented with confusion, irritability, and progressing memory concerns. At this time psychiatry was consulted to assist with diagnosis. Mr. P himself was an extremely poor historian. He was unaware of any memory problems or concerns by his physicians or family. The majority of the history was provided by his wife and medical record. Mrs. P stated her husband began to have difficulties with memory approximately two years previously, when she noticed forgetfulness and changes in personality. He became less social, irritable, intrusive, disorganized, and disoriented to time and place. He was no longer able to work as a result and was unemployed for the past two years. She describes the progression to be gradual and a fairly rapid decline. Extensive neurological workup did not reveal neurological illness as a result of infection, white matter disease, vascular disease, seizure, or tumor. The diagnosis of dementia using DSM-IV criteria was provided after review of the data already acquired and interview of patient, family, and neurology staff. Laboratory data reviewed included MRI, CT scan, serology, lumbar puncture, infectious workup and prior neuropsychological testing. With the early onset and rapid decline of cognitive function, appropriate diagnosis was necessary for treatment and prognosis. The consultant psychiatrist was also a memory disorders clinician and was aware that a PET scan recently had been approved for the assessment of dementia. This exam was utilized in this case and was suggestive of dementia of Alzheimer's type. Mr. P and his family were transitioned to memory disorders clinic for further evaluation of the severity of his cognitive dysfunction. Repeat neuropsychological testing was compared to prior exam. This demonstrated significant decline in the past six months. The Clinical Dementia Rating Scale was used to determine the stage of impairment.⁵⁵ Mr. P and his family were provided with the final diagnosis of dementia of Alzheimer's type severe stage. Appropriate pharmacological, psychological, and social supports and interventions were provided to the family.

The clinician used the DR.SEE paradigm. Data were gathered using records, other physicians, and the patient's wife because the patient was a poor historian and had serious memory problems. The clinician used the Clinical Dementia Rating Scale to determine the stage of cognitive impairment. External validators were used including MRI, CT scan, lumbar puncture, and PET scan. The experience of the psychiatrist, neurologist, and neuropsychologist were utilized in making a valid diagnosis.

Case three. The patient was a 42-year-old man who presented with feeling “depressed” for the past six months or longer. He felt like he had no energy and his wife esd concerned he was depressed. He went to work but felt he was having difficulty concentrating on his job. When he got home at night, all he wanted to do was lie on the couch and watch TV. He would fall asleep easily, but his sleep was fragmented. He noticed a steady increase in his weight and a decrease in his libido. Although he was not suicidal, he admited to feeling increasingly worthless about his current state. He was diagnosed with having a mild episode of major depression and was given a trial of an antidepressant.

During his follow-up visits there was no evidence of improvement in the patient's symptoms. His wife joined him on his third follow-up appointment, and during routine questioning about sleep, she made a comment about his “horrible” snoring and how she often would sleep in another room because of it. With this element of history, a diagnosis of possible obstructive sleep apnea was entertained, and upon examination of his oral airway, there was obvious crowding by the tongue and soft palate. His neck circumference was 18 inches. An Epworth Sleepiness Scale (ESS) was administered, and he had a significantly elevated score of 20.⁵⁶ The examination findings and Epworth Sleepiness Scale (ESS) score supported the working diagnosis of obstructive sleep apnea. A polysomnographic study was performed and the patient had an apnea-hypopnea index of 40. This provided the objective evidence for the diagnosis of sleep apnea. He was treated with C-PAP therapy, and within days noted significant improvement in his fatigue and sleepiness, as well as his concentration, libido, and feelings of self esteem. The clinician used the DR.SEE paradigm. Data were gathered by interviewing the patient. Important data were gathered from the patient's wife about his sleep problems. Physical examination of the neck, tongue, and palate also provided important data for the diagnosis. The clinician administered the Epworth Sleepiness Scale (ESS). External validators included a polysomnographic study. The experience of the neuropsychiatrist was utilized in making the valid diagnosis of sleep apnea and providing successful treatment.

Conclusions

Construct validity, consisting of validity criteria, is the core of psychiatry. The authors encourage clinicians to use as many validity criteria as possible to improve the validity of their diagnosis. Researchers and clinicians should utilize construct validity to revisit and redefine content validity of psychiatric disorders.