The Checkered History of American Psychiatric Epidemiology

Abstract

Context

American psychiatry has been fascinated with statistics ever since the specialty was created in the early nineteenth century. Initially, psychiatrists hoped that statistics would reveal the benefits of institutional care. Nevertheless, their fascination with statistics was far removed from the growing importance of epidemiology generally. The impetus to create an epidemiology of mental disorders came from the emerging social sciences, whose members were concerned with developing a scientific understanding of individual and social behavior and applying it to a series of pressing social problems. Beginning in the 1920s, the interest of psychiatric epidemiologists shifted to the ways that social environments contributed to the development of mental disorders. This emphasis dramatically changed after 1980 when the policy focus of psychiatric epidemiology became the early identification and prevention of mental illness in individuals.

Methods

This article reviews the major developments in psychiatric epidemiology over the past century and a half.

Findings

The lack of an adequate classification system for mental illness has precluded the field of psychiatric epidemiology from providing causal understandings that could contribute to more adequate policies to remediate psychiatric disorders. Because of this gap, the policy influence of psychiatric epidemiology has stemmed more from institutional and ideological concerns than from knowledge about the causes of mental disorders.

Conclusion

Most of the problems that have bedeviled psychiatric epidemiology since its inception remain unresolved. In particular, until epidemiologists develop adequate methods to measure mental illnesses in community populations, the policy contributions of this field will not be fully realized.

This article considers the changing relationship of psychiatric epidemiology, psychiatric treatments, and social policy. Ideally, the goal of epidemiological studies is to establish causal relationships between the environment and patterns of illness. In contrast to much of epidemiology, however, psychiatric epidemiology has done little to illuminate either the etiology or the environmental factors that create mental illnesses. Because of the limits of its knowledge base, the policies stemming from epidemiological studies have served to justify the predominant ideological and administrative interests of psychiatrists, researchers, and policymakers. The lack of an adequate system of classification of mental illnesses has been the primary factor limiting the development of psychiatric epidemiology and, consequently, its usefulness for developing effective public policy toward mental illnesses.

The Nineteenth-Century Background

In the nineteenth century, a new type of quantitative social inquiry emerged that became the eventual basis for social policy. Underlying this commitment was the assumption that such a methodology could illuminate and explain social phenomena. A prerequisite was a classification system capable of ordering a seemingly infinite variety of statistical data. Hence, a preoccupation with classification was accompanied by the collection of statistical data dealing with a variety of subjects.

That epidemiology began to take its modern form in such an environment was hardly surprising. Building on the medical topographies of the late eighteenth century, the overriding objective of the new epidemiology was to use statistical data to illuminate the relations between morbidity and mortality patterns, on the one hand, and broad environmental factors, on the other. Initially, epidemiology was concerned with the geography of disease; subsequently its attention shifted to the distribution of disease in the population and the identification of causal relationships between specific infectious diseases and external determinants. The development of appropriate human interventions would presumably follow, thus both elevating the health of the population and modifying inappropriate behavioral patterns. Perhaps the best-known example is John Snow's (1854) famous study of the cholera epidemic in mid-nineteenth-century England, which determined that a contaminated water supply was the vehicle transmitting the disease. Such work laid the basis for governmental interventions to limit the epidemic. In this respect, epidemiology had a crucial role in shaping public policy.

It would not have been surprising if psychiatry had been at the forefront of nineteenth-century epidemiology. From the very creation of their specialty in the 1820s, American alienists (i.e., psychiatrists) assiduously collected statistical data. The annual reports of their mental hospitals, as well as the American Journal of Insanity (which commenced publication in 1844 and became the American Journal of Psychiatry in 1921) included statistics on the demographic and geographical characteristics of patients with severe mental disorders as well as on the results of therapy. The federal census of 1840 also collected aggregate data on mental illnesses. Nevertheless, the psychiatric preoccupation with statistics was far removed from the emerging discipline of epidemiology. Rather, nineteenth-century psychiatrists were concerned with data related to a large institutionalized population of persons with severe mental illnesses. Statistics were useful because they provided evidence of the benefits of institutional care. At the time, psychiatrists were not interested in causal mechanisms of particular mental illnesses largely because of the absence of a reliable and valid system of classifying mental illnesses. Instead, their major concerns were with the policy implications of data concerning institutional populations, not with epidemiology more generally. Nor did they make any connection between statistical data and their therapeutic systems.

Although nineteenth-century psychiatrists—virtually all of whom were employed in public and quasi-public institutions—were avid data collectors, they were not epidemiologists. Instead, they used statistics mainly to demonstrate high curability rates among institutionalized populations. They also used statistics for purposes of policy advocacy (Grob 1973). In 1850, for example, Edward Jarvis demonstrated that there was an inverse relationship between the distance to and the use of a mental hospital. The lesson was clear: public policy should not favor large institutions serving a wide geographical area but, rather, smaller and geographically dispersed ones (Jarvis 1850). Nowhere was the descriptive, as compared with the analytic, character of the nineteenth century better illustrated than in Jarvis's pioneering study of insanity in Massachusetts in 1854. Jarvis conducted a census that identified everyone with a mental illness and prepared a report that in printed form exceeded two hundred pages and was replete with statistics on each person's geographical location, ethnicity and economic status, sex, and race. Reflecting the social and ethnic tensions characteristic of the Bay State, he found an association between poverty and insanity; both were traceable to an “imperfectly organized brain and feeble constitution” (Grob 1971, 56). Foreigners (i.e., the Irish) were particularly susceptible to insanity because they were ill adapted to the physical and cultural conditions of American society, to say nothing of their intemperance (Grob 1971, 45–62).

That epidemiological analysis was lacking was not surprising. Nineteenth-century American psychiatry was a managerial, administrative, and clinical specialty. Although psychiatrists subscribed to a somatic interpretation of insanity, they nevertheless accepted an environmental etiology. They adhered to a version of “moral treatment,” a therapy developed in France by Philippe Pinel and in England by William Tuke. This system of therapeutics (which included drugs, emetics, and tonics as well) was based on the belief that institutionalization was required to treat the insane because it allowed psychiatrists to create a new environment that broke with those prior harmful environmental influences that led to insanity. Consequently, the imperatives of the hospital as a social system reinforced managerial concerns and left little time for more abstract and theoretical questions regarding mental illnesses. Indeed, the therapeutic systems of the early and mid-nineteenth century bore little relationship to statistical data; they reflected instead religious and moral values.

To be sure, mid-nineteenth-century psychiatrists believed that there was a direct relationship between what they perceived was the rising incidence of mental illnesses and the advance of civilization. Such beliefs, however, did not arise out of the application of an epidemiological methodology but, rather, reflected social and religious values. Jarvis went the furthest of any of his colleagues in dealing with the assertion that insanity was increasing. He noted that the use of institutional data could not prove this claim. But he insisted that changing social and economic structures and moral behaviors explained the increase in the incidence of mental disorders. Modern society, he wrote, brought

more opportunities and rewards for great and excessive mental action, more uncertain and hazardous employments, and consequently more disappointments, more means and provocations for sensual indulgence, more dangers of accidents and injuries, more groundless hopes, and more painful struggle to obtain that which is beyond reach, or to effect that which is possible. (Jarvis 1852, 363–64)

The interaction of these variables was related to changing patterns of incidence (Jarvis 1852). As Isaac Ray (perhaps the most important nineteenth-century American psychiatrist) noted in 1852, insanity was “the price we pay for civilization” (Butler Hospital for the Insane 1852, 19).

One of the greatest barriers to epidemiological inquiry was the status of psychiatric nosology. Epidemiology requires clear classes or categories of disease in order to facilitate the collection of statistical data. Prevailing nineteenth-century psychiatric nosologies, however, did not lend themselves to any kind of precision. Instead, the diagnostic categories—which included mania, monomania, melancholia, dementia, and idiocy—were based on symptoms. Categories based on symptoms, explained Amariah Brigham (superintendent of the Utica State Lunatic Asylum and editor of the American Journal of Insanity), “must be defective, and perhaps none can be devised in which all cases can be arranged” (Utica State Lunatic Hospital 1843, 36). The links between organs and behavior were shrouded in mystery. Psychiatrists therefore attempted to identify the presence of disease by observing external signs or symptoms. In this respect, they differed but little from other physicians. Before the specific germ theory of disease, most physicians defined pathological states in terms of external and visible symptoms, such as fever. But for psychiatrists, the problem was even more complex, as behavioral symptoms were amorphous and far more difficult to classify than even fevers. Thus, most psychiatrists understandably conceded that their nosologies were, to all intents and purposes, nearly useless.

Origins of Psychiatric Epidemiology

Curiously enough, the impetus to create an epidemiology of mental disorders came largely from outside psychiatry or medicine. Toward the end of the nineteenth century, emerging social science disciplines were concerned with developing a scientific understanding of individual and social behavior and applying it to a series of pressing social problems. In their quest for empirical data, they drew on the tradition of statistical analysis associated with such figures as Adolphe Quetelet (1842/2010). Ultimately, a general consensus developed on the indispensable utility of the federal census. To social scientists, the census was not merely an instrument to collect data or to apportion representation in the United States Congress. On the contrary, the census represented a radical faith that when merged with administrative rationality, quantitative research could replace politics (Grob 1978; Porter 1986). Statistical knowledge could thus serve as the foundation for social policy and end the pernicious bickering over theory, principles, and politics.

After 1870, the federal census underwent fundamental changes. The census of 1870 included data on the insane aggregated by race, age, and sex. The major change then came with the tenth census in 1880, which included a 581-page volume dealing with the “defective, dependent, and delinquent classes.” Prepared by Frederick H. Wines (a major figure in late-nineteenth-century welfare policy), the document (with psychiatric assistance) aggregated the statistics of insanity under seven categories: mania, melancholia, monomania, paresis, dementia, dipsomania, and epilepsy. For the most part, such categories were functional and descriptive, permitting psychiatrists to organize material and communicate with one another but not dealing with etiology (U.S. Bureau of the Census 1888).

The 1890 census, which included a 755-page volume on dependency and dependent groups, followed the precedent that Wines set (U.S. Bureau of the Census 1895). In 1904 and 1910 the Census Bureau conducted two special censuses dealing with the institutionalized insane. The former focused attention on the ethnic and racial characteristics of the institutionalized insane, a reflection of the growing fear that the integrity of the native-born population was being threatened by the large-scale immigration of undesirable groups from eastern and southern Europe (U.S. Bureau of the Census 1906). The special census of 1910, however, reflected a growing sophistication in the use of statistical data. It cast doubt on the belief that insanity was increasing. By correcting for the age distribution of the entire native- and foreign-born population, the census also demonstrated that the claim that immigrants had far higher rates of mental illnesses was erroneous (U.S. Bureau of the Census 1914). To be sure, the modern federal censuses that emerged in the late nineteenth and early twentieth century were not designed as instruments for epidemiological inquiry. Nevertheless, they laid the foundation for the discipline in an important way. Together with the collection of statistical data, their diagnostic categories—admittedly imprecise and flawed—were nevertheless indispensable for the future development of psychiatric epidemiology.

Within psychiatry, the traditional preoccupation with the institutionalized insane began to give way to a new vision. As Thomas W. Salmon (medical director of the National Committee for Mental Hygiene) argued in 1917, psychiatrists had to reach beyond the boundaries of the mental hospital and play a crucial role “in the great movements for social betterment” (Salmon 1917, 98). They had to lead the way in research and policy implementation, lay the foundations of mental hygiene, supervise the care of the retarded, promote eugenics, control alcoholism, manage abnormal children, define the treatment of criminals, and play crucial roles in the prevention of crime, prostitution, and dependency (Grob 1983).

Such concerns reinforced psychiatric interest in the kind of statistical analysis characteristic of the social sciences. Yet the chaotic state of data collection as well as the absence of a uniform nosology posed a seemingly insurmountable barrier. In 1917 the Committee on Statistics of the American Medico-Psychological Association (whose origins dated back to 1844 and which in 1921 was renamed the American Psychiatric Association) urged that all mental hospitals adopt a uniform reporting system. “The present condition with respect to the classification of mental diseases is chaotic,” the committee observed, and this “condition of affairs discredits the science of psychiatry and reflects unfavorably upon our Association” (Committee on Statistics 1917, 256).

It was evident that if the Census Bureau was to continue its data-collecting activities, it would have to pay far greater attention to the diagnostic categories it employed. With the cooperation and assistance of the National Committee for Mental Hygiene, the American Medico-Psychological Association produced the first uniform nomenclature of mental diseases in 1918 (Committee on Statistics 1918; Statistical Manual 1918). Horatio M. Pollack, a statistician employed by the New York State Hospital Commission and subsequently the director of mental hygiene statistics of the State Department of Mental Hygiene, played a key role in creating the new nosology. He and his colleague Benjamin Malzberg were important figures in fostering the growing relationship between psychiatry and social science and creating the foundations of psychiatric epidemiology.

For the Census Bureau, the adoption of a formal classification system was of major significance. By 1920 the bureau had compiled its own nomenclature of diseases, which included psychiatric diseases. It conceded its inability to draw a clear line between symptoms and diseases and admitted that in the future, many of the terms used to describe a pathological condition could be considered symptoms if the etiology of the disease was discovered. Nevertheless, there was no alternative (U.S. Bureau of the Census 1920). Three years later, the Census Bureau began its annual collection of the statistics of the patients in mental hospitals (U.S. Bureau of the Census 1926). A new era of psychiatric epidemiology was about to begin.

Psychiatric Epidemiology Moves from Institutions to the Community

The sole concern with institutional populations and the lack of an adequate classification system meant that a true psychiatric epidemiology did not emerge for many years after the field of epidemiology more generally. Then, when psychiatric epidemiology finally began to take shape in the early twentieth century, social scientists rather than psychiatrists determined its basic character. But because most social scientists were not trained in medicine, they gave little thought to the formidable problems posed by a nosology based on symptomatology rather than etiology (paresis and pellagra being notable exceptions). Thus, statisticians and social scientists tended to uncritically accept psychiatric classifications and then analyze patient data through such categories as age, sex, mental condition, race, occupation, education, place of residence, and ethnicity. The relevance of such categories to an understanding of the nature and etiology of mental illnesses remained unclear, if only because correlation cannot be equated with causality.

Yet these demographic data about institutionalized persons with mental illnesses were useful to policy officials concerned with present and future trends and planning. Preoccupation with policy issues, however, reinforced the lack of attention to the imprecise nature of diagnostic categories, an oversight that contradicted accepted epidemiological methodology. In this sense, traditional epidemiology—which from its beginnings in the nineteenth century had focused on discrete infectious diseases—differed somewhat from the psychiatric preoccupation with descriptive categories.

In 1911 the New York State Hospital Commission (later the Department of Mental Hygiene) expressed the hope that at some point light would be shed on the etiology of mental disorders. At the same time it noted that it would be wrong to overlook the significance of economic and sociological studies. It listed a series of subjects worthy of investigation: the role of immigration, the “effect of the mingling of races,” and the relation of insanity to occupation, housing, conjugal relations, and other environmental factors (New York State Hospital Commission 1912–1913). During the 1920s and 1930s, data collected by the Bureau of the Census, the New York State Department of Mental Hygiene, and the Massachusetts Commissioner of Mental Diseases served as the raw material of psychiatric epidemiology. Horatio Pollock published numerous studies dealing with the expectation and outcome of mental diseases and the relationship of insanity to environment, sex, and age, as well as analyses of therapeutic effectiveness and outcomes (Pollock 1941). Benjamin Malzberg, who joined Pollock's office in 1927 and became its chief in 1944, wrote dozens of articles and books analyzing the relationship between mental disorders and age, life expectancy, ethnicity, race, religion, birth order, literacy, and economic factors. His influential study (in collaboration with Everett S. Lee) of migration and mental disease demonstrated that natives born in New York State had significantly lower rates of psychoses than did natives who were born elsewhere and had migrated to the state. Rates for foreign born were usually intermediate between the two. Using first admissions data, he also found that African Americans had higher rates of mental diseases than whites did but that the differential decreased between 1940 and 1950. He attributed the difference to environmental influences and the fact that African Americans had been a migratory population (Malzberg 1940, 1959, 1960; Malzberg and Lee 1956). The examples of Pollock and Malzberg had parallels elsewhere (Dayton 1940).

The availability of such statistical data seemed to herald the emergence of a mature psychiatric epidemiology. In 1938 Henry M. Elkind, medical director of the Massachusetts Society for Mental Hygiene, asked whether there was an epidemiology of mental disease. He conceded that psychiatric categories and data involved “more or less intangible concepts, because psychiatry, dealing so much with the mind, must perforce concern itself with subjective experience, which is so difficult for scientific study and analysis.” Moreover, the various schools of psychiatric thinking differed on the question of etiology, although this was characteristic of medicine generally. Nevertheless, Elkind insisted that mental disease was a proper concern of epidemiology (Elkind 1938, 248).

That same year, 1938, Carney Landis and James D. Page published their influential Modern Society and Mental Disease. Trained in psychology, they distinguished between the biosocial and the medical approaches. The latter was concerned with the individual patient; the former with those factors having to do with the “incidence, hospitalization, development, and control of mental disease in the general community.” In an analysis of admission, readmission, discharge, and death rates in state mental hospitals, they concluded that the data favored “the argument that the basic etiological factors of ‘mental’ disease are physiological and constitutional rather than psychological.” Although not opposed to eugenic sterilization, they doubted its effectiveness. Nor was there much promise for a reduction in the rates of senile dementia or cerebral arteriosclerosis. The solution to the problem of mental disease, they concluded, lay in “basic medical and biological research” (Landis and Page 1938, 3, 151, 160).

By this time, the traditional preoccupation with institutional populations had begun to be superseded by studies that analyzed mental illnesses in socioenvironmental terms and their incidence in the community. Aaron J. Rosanoff had undertaken such a study in 1916, even though he was aware of major methodological problems (Rosanoff 1917). Nevertheless, socioenvironmental and community studies remained the exception rather than the rule. The publication in 1929 of Robert and Helen Lynd's classic Middletown, however, awakened interest in community studies generally (Lynd and Lynd 1929). One of the earliest and most famous works of this sort was Robert E. L. Faris and H. Warren Dunham's Mental Disorders in Urban Areas, which attempted to demonstrate a relationship between different types of hospitalized psychoses and certain community conditions (Faris and Dunham 1939). Although a number of ecological studies of mental illnesses had been undertaken in the 1930s, none compared with the Faris and Dunham analysis (Cohen and Fairbank 1938; Cohen, Fairbank, and Greene 1939; Cohen, Tietze, and Greene 1939).

To move from a concern with mental illnesses in institutionalized populations to its incidence in the general population and the role of socioenvironmental elements represented an extraordinary intellectual leap. The earlier studies of institutionalized patients represented a logical step to gather data that would serve as a guide for public officials responsible for mental health policy. But the study of incidence and socioenvironmental factors was an entirely different matter, although policy concerns were not irrelevant. Given the descriptive and imprecise nature of psychiatric nosology, how was it possible to identify, with any degree of precision, either persons with mental illnesses in the community or the relevant socioenvironmental influences? This and other fundamental questions were largely ignored, partially because the earlier preoccupation with heredity had given way to environmental modes of thinking in American social thought in general and the social sciences in particular (Degler 1991; Grob 1991).

Psychiatric epidemiology was largely the creation of a group of statistically oriented social scientists concerned with problems relating to poverty, dependency, and welfare. They realized that institutional populations were notoriously poor sources for epidemiological inquiry. Psychiatric diagnostic categories were neither reliable nor valid. Moreover, the study of institutional populations was not a measure of incidence or prevalence. Instead, it often reflected such factors as the availability of hospital beds, concerns related to the willingness of individuals and their families to define conditions as mental illnesses, and the community's tolerance for deviant behavior. As Robert H. Felix and Robert V. Bowers pointed out in 1948, researchers using such sources of data had “no control over the case-finding process, over the record keeping, or even the diagnosis.” Erratic patterns of institutionalization, unstandardized records, and the widely varying training and skill of the mental hospital staff in classifying diseases rendered earlier studies virtually useless. Indeed, they concluded, “studies have not always been made with much perception of sound methodological principles.” Felix and Bowers called instead for ecological studies of the etiology of mental disease that focused on socioenvironmental factors. They were especially sympathetic to the work of such individuals as Robert S. and Helen M. Lynd, W. Lloyd Warner, Karen Horney, and Erich Fromm and insisted that “the field of the relation between personality and socioenvironmental factors is providing intriguing insights into the etiology of mental disorders” (Felix and Bowers 1948, 127–28, 134). Oddly enough, they did not recognize that the methodological problems of isolating the role of socioenvironmental factors in the molding of personality, using the general population as a base, far exceeded the methodological problems of studying only institutionalized persons with serious mental illnesses. Moreover, the absence of clear diagnostic categories rendered associations between socioenvironmental factors ever more problematic.

The experiences of military psychiatrists during World War II provided a major turning point in the focus of psychiatric epidemiology (Grob 1991). These psychiatrists’ wartime experiences seemed to indicate the importance of intensely stressful environments for rates of mental illnesses. Psychiatric casualties soared among soldiers who were seemingly normal before entering combat. The policy lesson seemed clear: environmental conditions rather than individual predispositions were the primary causes of mental disorder. The association of highly stressful environments with massive amounts of mental distress prepared the way for the social emphasis that came to dominate psychiatric epidemiology during the 1950s and 1960s.

After World War II, community and demographic studies of persons with mental disorders proliferated, with changes in data-gathering capabilities playing a significant role. In 1946 the Bureau of the Census conducted its final annual census of patients in mental hospitals (U.S. Bureau of the Census 1948). As a result of the passage of the National Mental Health Act that same year, responsibility for such data-gathering activities was transferred to the Public Health Service and the soon-to-be-created National Institute of Mental Health (NIMH), which eventually established a Division of Biometry and Epidemiology. The statistical activities of the NIMH reinforced the traditional preoccupation with demographic variables. At the NIMH, Morton Kramer set up the Model Reporting Area for mental health statistics in order to improve data dealing with the care of persons with mental illnesses. During the 1950s and 1960s, Kramer and his colleagues produced a series of studies on patient outcomes and evaluations of community mental health programs (Ellenberg 1997).

The major epidemiological studies of the immediate postwar era attempted to measure incidence, to define with precision the role of socioenvironmental variables, and to deal with the relationship between social class, diagnosis, treatment, and mental disorders. Typical of postwar studies were such works as Goldhamer and Marshall's Psychosis and Civilization and Hollingshead and Redlich's Social Class and Mental Illness (Goldhamer and Marshall 1953; Hollingshead and Redlich 1958). The former sought to link social stratification with the prevalence (and, to a lesser extent, the incidence) of mental illnesses and to show how social class was related to therapy.

Others took a more global approach. The famous Midtown Manhattan Study, conceived by Thomas A. Rennie in 1950 and continued after his death in 1956 by colleagues, was an intensive study of an area with a population of about 174,000. Ultimately, the project used a sample of 1,911 individuals between the ages of twenty and fifty-nine, of whom 1,660 completed a long and detailed questionnaire. Clinicians rated the responses according to the severity of psychiatric symptoms and the degree of impairment and then related their ratings to a range of demographic data. The survey found that only 18.5 percent were free of symptoms; 36.3 percent had mild symptoms; 21.8 percent had moderate symptoms; and 23.4 percent had severe symptoms and were often incapacitated. The study also found that mental health was in large measure a function of socioeconomic status (Srole et al. 1962).

At the same time, the Milbank Memorial Fund decided to move beyond the awarding of grants, and in 1949 it sponsored a conference dealing with the epidemiology of mental disorder, especially the etiological role of socioenvironmental variables. All the participants at the conference—including such figures as Ernest M. Gruenberg and Alexander Leighton—played important roles in stimulating interest in psychiatric epidemiology during and after the 1950s (Milbank Memorial Fund 1950, 1953). To be sure, although many of these epidemiological studies differed sharply in design and methodology, their appearance alone indicated that interest in psychiatric epidemiology was on the rise.

The growing significance of epidemiology in the late nineteenth and twentieth centuries rested on the prior identification of disease as a biological process and the manner in which it affected individuals in the population. As Richard J. Plunkett and John E. Gordon wrote in their critique of psychiatric epidemiology for the Joint Commission of Mental Health and Illness (1955–1961), “Knowledge of the principles governing disease as a mass phenomenon, and the techniques for its study arose from observations of infections—superb ecologic examples of one organism reacting with another in a common environment” (Plunkett and Gordon 1960, 15). From that point, the epidemiologist could move on to the larger ecological problem and study the social and environmental elements that retarded or promoted the disease, the number and types of people it affected, and its geographical distribution and frequency.

At a more fundamental level, psychiatric nosologies—with only isolated exceptions—rested on a descriptive rather than an etiological foundation. For centuries, most forms of mental illnesses defied attempts to create definitive and universally recognized diagnostic categories. “When observers are unable to agree what they are looking at,” noted Plunkett and Gordon, “they are likely not to have a clear conception of its origin. In the absence of a common descriptive language, efforts to discover the etiology of all but a few organic mental diseases have met with ill success” (Plunkett and Gordon 1960, 5). Indeed, the absence of an adequate nosology rendered existing data on prevalence and incidence virtually useless. Consequently, the results of psychiatric epidemiological studies often differed from studies of infectious diseases precisely because of the variations in the design of studies and classification systems as well as the subjective observations of the investigators themselves.

Generally speaking, there was a distinct tendency among psychiatrists and social scientists to overlook or ignore such methodological problems. In 1970 Paul Lemkau extolled population studies of incidence because they substituted more accurate “operational definitions of mental illnesses” in place of “concepts of often ill-defined syndromes” used by psychiatrists (Lemkau 1970, 119). Population surveys, he noted, led to recognition of the association between social class and the prevalence of mental illness (Lemkau 1970). Aside from the fact that the line of demarcation between symptoms and syndromes was extraordinarily vague, there remained the additional problem of explaining the relationship between social class and mental illnesses on other than a statistical basis. Indeed, John Cassell's observation—that if social-class findings were reversed, it would still make little difference to an understanding of the illness—was by no means an irrelevant criticism (Cumming 1970).

Despite the difficulties of showing causal connections between stressful social environments and associated rates of mental illnesses, the findings from psychiatric epidemiology contributed to the NIMH's primary focus on the socioeconomic determinants of psychological conditions. Federal policies assumed that changing environmental factors such as poor housing, poverty, and crime rates would be the most effective way to lower levels of mental illnesses in the population. By the late 1960s, however, this focus had become a source of serious trouble for the NIMH.

The Movement to Categorical Illnesses

The face of psychiatric epidemiology radically changed in the late 1970s and early 1980s. This revolution largely reflected a response to the challenges that the psychiatric profession and the NIMH faced in maintaining their legitimacy. The profession's diagnostic manual, the psychodynamically oriented DSM-II (APA 1968), contained only cursory descriptions of its entities. Neither research-oriented psychiatrists nor epidemiologists could employ the definitions in this manual. Instead, the latter relied on general measures such as the Langner (1962) scale, which did not reflect particular types of mental disorder but, instead, a variety of depressive, anxious, and psychosomatic symptoms. Such scales produced continuous measures of “distress” that were not readily translatable into any particular diagnosis. The inadequacies of this classification system contributed to highly publicized criticisms that psychiatry was unable to define even its most central disease entities (e.g., Rosenhan 1973; Szasz 1974). Challenges to the DSM-II diagnostic system also arose from within the profession. Research indicated that the reliability of psychiatric diagnosis was appallingly low. For example, the prominent US-UK Diagnostic Project asked American and British psychiatrists to diagnose the same cases. It found that American psychiatrists were far more likely to diagnose schizophrenia in the same instances in which British psychiatrists diagnosed manic depression (Kendell, Cooper, and Gourlay 1971). Psychiatry's diagnostic system thus seemed to require a fundamental overhaul to bolster confidence in the idea that psychiatrists actually studied and treated genuine disease entities.

The NIMH was undergoing its own crisis in the late 1960s and early 1970s. Since the 1950s, the agency had emphasized the study of general personality, developmental, and social issues more than specific types of mental illness. Accordingly, it awarded 60 percent of its grant funding to psychologists and social scientists and less than 40 percent to psychiatrists and other medical and biological scientists (Grob 1991, 66–67). After Richard Nixon became president in 1968, his administration and the Congress began to attack the NIMH for sponsoring research on broad social problems like poverty, racism, and violence. Although social problems research accounted for only around a fifth of the institute's portfolio, it was a lightning rod for attacks on its overall mission. Similarly, major NIMH-sponsored epidemiological studies focused on the fundamental importance of socioeconomic conditions as determinants of mental health (e.g., Leighton et al. 1963; Srole et al. 1962). By the early 1970s, psychosocial research had become a political liability, and funding for the NIMH was declining at a rate of about 5 percent each year (Baldessarini 2000; Wilson 1993).

At the same time, the treatment system for mental illness was undergoing radical changes. Until the 1950s, it had been bifurcated into a large inpatient sector and a smaller community-based segment focused on dynamic psychotherapy (Grob 1991). The development of the major and minor tranquilizers and an expanding federal role in mental health policy led to a widespread movement of patients from institutions into the community that began in about 1955 and accelerated during the 1960s and 1970s. A growing need to know how many people in the population needed mental health services and an intensification of drug treatments as the first line of response to mental illness accompanied this deinstitutionalization.

By the late 1970s, biologically oriented researchers had joined the fight against psychosocial research at the NIMH. This group was deeply concerned that research on social problems and general psychosocial conditions would damage the institute's reputation and subject it to a backlash against all its research programs (Kolb, Frazier, and Sirovatka 2000). These researchers argued that the study of specific mental disorders would both enhance the quality of scientific research and justify the institute's mission in the face of political opposition.

During the same period, the American Psychiatric Association (APA) appointed a prominent research psychiatrist, Robert Spitzer, to lead a task force charged with developing a new edition of its classification manual, the Diagnostic and Statistical Manual, 3rd ed. (DSM-III). Although the APA itself did not envision a radical change in the manual, Spitzer created a diagnostic revolution. His influence led to the establishment of reliable, symptom-based definitions and the purge of etiological assumptions from each diagnosis to become the guiding principles of the manual. To accomplish these goals, Spitzer and the various task forces he appointed to develop measures of the various classes of disorders began with the Feighner criteria. These were a set of fourteen descriptive classifications including depression, anxiety, and phobic disorders, which were developed by a group of research psychiatrists at Washington University in St. Louis, led by Samuel Guze and Eli Robins and named after the psychiatric resident in the department who was the first author of the initial publication (Feighner et al. 1972). Spitzer collaborated with the Washington University group to develop the research diagnostic criteria (RDC), which expanded the Feighner criteria into twenty-five major diagnostic categories and numerous subtypes of these categories (Spitzer, Endicott, and Robins 1978). In highly modified forms, the Feighner criteria and the RDC became the intellectual foundation for the symptom-based entities of the DSM-III, which was published in 1980.

The new generation of epidemiological studies resulted from the close collaboration of prominent research psychiatrists, their social scientific allies, and the NIMH. In the mid-1970s, the NIMH prioritized research that would simultaneously serve as the basis for a reliable system of diagnostic entities for use in psychiatric research and treatment and for a new type of epidemiological research that was grounded in the same specific diagnoses. The agency sponsored the development of a data-collection instrument that could translate the major diagnostic categories of the manual into standardized measures of the major mood, anxiety, and substance abuse disorders that would serve as the basis for epidemiological studies of the treated and untreated prevalence and incidence of the major DSM-III diagnostic entities. The Washington University group had also developed the Renard Diagnostic Interview (RDI), an instrument based on the Feighner criteria, which lay interviewers could use to generate diagnoses in community-based populations (Weissman, Myers, and Ross 1986). Working with Eli Robins and his wife, the epidemiologist and sociologist Lee Robins, Spitzer used the RDI as the basis for the RDC, which in turn served as the foundation for the Diagnostic Interview Schedule (DIS). The initial epidemiological study using the RDC, conducted in New Haven, Connecticut, in the mid-1970s, indicated that trained laypersons could administer standardized interviews that would generate specific diagnoses (Weissman and Myers 1978).

After considering a variety of possible instruments to measure mental disorder in the community, the NIMH selected the DIS because it most closely reflected the categories of the emergent DSM-III (Robins 1986). The DIS could be used to measure a number of the specific diagnostic conditions in community populations that were supposed to be comparable to the major DSM-III categories. For example, the criteria for major depressive disorder asked respondents if they had ever experienced a two-week period of dysphoric mood accompanied by four or more symptoms, such as loss of appetite, sleep disturbance, fatigue, and trouble concentrating. The style of psychiatric epidemiology that emerged with the DIS thus reflected a mirror image of psychiatry's new classification system.

The diagnostic categories of the DSM-III were based on overt symptoms rather than any underlying etiological or contextual factors. This decision permitted epidemiologists, with little change, to apply diagnoses developed for clinical patient populations to surveys of the general population. The symptom-based nature of the DSM-III categories also enhanced the ability of surveys to measure these diagnoses in a uniform and reliable way. The DIS used closed-format questions that were asked in precisely the same way and could lead to computer-generated diagnoses. Moreover, the standardization of the diagnoses minimized cost and maximized efficiency because laypersons needed only a few hours of training to administer the interviews.

The DIS transformed the face of psychiatric epidemiology. It was the basis of the first national study of the prevalence of specific mental illnesses in the community: the Epidemiologic Catchment Area (ECA) study launched in the early 1980s (Robins et al. 1984). The idea of a national study of psychiatric epidemiology was itself a radical departure from previous efforts. Earlier studies had focused on particular communities, whether Chicago, Manhattan, New Haven, or rural Nova Scotia. They were thus highly attuned to the particular social contexts in which disorders arose. By their nature, studies that attempted to establish nationwide incidence and prevalence rates focused on random samples of individuals rather than on community contexts. One result of this methodology was to turn psychiatric epidemiology away from the study of social circumstances toward that of individuals.

The ECA surveyed more than 18,000 adults in the community and 2,500 persons in institutions in five sites (New Haven, Durham, Baltimore, St. Louis, and Los Angeles) to generate national estimates of prevalence. The second major community study of the prevalence of specific psychiatric disorders relying on the DSM diagnostic categories was the National Comorbidity Survey (NCS), which the NIMH fielded in 1991 with a ten-year follow-up begun in 2001 (Kessler et al. 1994, 2005). The NCS, a sample of about 8,100 persons meant to represent the population of the United States, used the Composite International Diagnostic Interview (CIDI), an instrument similar to the DIS but with a different pattern of asking questions (Regier et al. 1998).

The ECA estimated that about 16 percent of the population had at least one current psychiatric disorder and that about 20 percent had had some disorder over the past year. About a third reported a lifetime history of disorder. When the results of a second ECA survey conducted with the same subjects one year after the original survey were taken into account, the estimates of lifetime prevalence increased from 32 percent to 44 percent of the population (Regier et al. 1998).The initial NCS produced even higher rates of mental disorder, estimating that nearly half—48 percent—of people had some lifetime disorder and that 29 percent of the population had had some disorder over the past year. The results from the NCS restudy conducted in the early 2000s showed comparable findings. Overall, about 29 percent of the population experienced an anxiety disorder, 21 percent a mood disorder, 25 percent an impulse control disorder, and 15 percent a substance use disorder (Kessler et al. 2005). Moreover, epidemiological research seemed to show that rates of mental illnesses were growing at an alarming rate. One highly publicized study, for example, claimed that each successive age cohort born between 1915 and 1965 had growing rates of depression (Klerman 1988; Klerman and Weissman 1989). The major social correlates of mental disorder, including low income, poor education, and stressful life events, were identical to those found in the earlier generation of epidemiological studies.

Psychiatric epidemiologists generally agree that the ECA and NCS translation of the criteria of the DSM diagnostic categories for use in community populations has solved epidemiology's perennial problem of defining the attributes of a psychiatric case. In 1992, for example, Lee Robins wrote that “the problems of measuring prevalence have been largely solved. Diagnostic interviews have been developed which are based on official diagnostic criteria and are so well-standardized that they can be administered very reliably by lay interviewers and scored by computer” (Robins 1992, 1). The findings from the ECA and especially from the NCS are cited in virtually every research article about mental disorders. They are the basis for statements that huge and increasing proportions of the population suffer from mental disorders, which are now widely cited in the scientific and popular literature, in pharmaceutical advertisements, and in advocacy documents (USDHHS 1999). The principal investigator of the NCS, Ronald Kessler (a sociologist by training), is the world's most cited mental health researcher (http://www.uspreventivemedicine.com/About-Us/National-Advisory-Board/Ronald-Kessler.aspx). Finally, the results of these studies have contributed to a dramatic change in social policies regarding mental illness.

Current Psychiatric Epidemiology and Social Policy

A focus on the social environment dominated the initial epidemiological studies. While much of that research simply correlated rates of mental illnesses with such categories as age, sex, ethnicity, race, occupation, and education, it also reflected the view that socioeconomic conditions strongly affected these rates. With rare exceptions, epidemiological studies assumed that mental illnesses were rooted in social circumstances, not that mental illnesses caused these circumstances. Their underlying philosophy was that policy efforts that reduced inequality, enhanced opportunity, and improved living conditions provided the optimal response to the untreated cases of mental illnesses they uncovered in the community.

Epidemiological studies since 1980 retain the earlier emphasis on describing the prevalence of disorders and rates of treatment among members of different social characteristics. Nonetheless, the dominant ideology of the psychiatric profession and the NIMH had changed from the assumption that the social environment caused psychiatric disturbance to the view that mental illnesses reflected brain malfunctions that created social impairments. In addition, therapeutic orientations had moved from a focus on psychosocial treatments toward reliance on drugs. Reflecting these changes, epidemiologists, particularly those associated with the NCS, used the same correlational results as those found in earlier studies to reverse the assumption that social environments have causal influences on psychological well-being. Relying on retrospective data that stemmed from respondents’ recall of the age when their symptoms first arose, they inferred that mental disorders—especially anxiety and depression—had early ages of onset and later in the life course caused much social impairment, such as limited educational and occupational attainment, poor physical health, welfare dependency, and teenage pregnancy (Kessler 2003; Kessler et al. 1995, 1997, 2004, 2005).

These findings led to a transformation of policy recommendations from changing social conditions to identifying at an early age those individuals in need of professional mental health treatment and giving them drug and other therapies. To accomplish these ends, epidemiologists recommend widespread screening programs for psychiatric symptoms in schools and primary medical care settings. Adults can be reached through workplace programs that screen employees for signs of common mental illnesses, recruit them into treatment, and then provide them with drug and psychological therapies (Kessler 2002). Epidemiology turned away from its previous policy emphasis on changing social conditions that might give rise to vulnerabilities in the first place toward a focus on intra-individual factors.

In one sense, the recent policy thrust of psychiatric epidemiology continues the field's aim since the 1950s of ascertaining rates of untreated cases of mental disorder, with the ultimate goal of reducing or preventing rates of psychiatric disturbances. What is new is the assumption that mental illnesses precede and lead to social impairments. The reversal of the social policy implications of epidemiological studies and consequent focus on intra-individual rather than environmental considerations is congruent with the biological emphasis found in the current psychiatric profession, the NIMH, and pharmaceutical companies. The findings from these studies have not altered, but the change in their interpretation makes the research compatible with the dominant ideologies and interests of the most powerful groups that formulate mental health policy. Current policy almost exclusively focuses on professional interventions that change individuals (Lantz, Lichtenstein, and Pollack 2007). Indeed, even widely heralded studies that find considerably larger environmental than genetic effects nevertheless focus policy recommendations on using the genetic findings to tailor individual therapies (Caspi et al. 2003).

Problems with Recent Psychiatric Epidemiology

Despite the widespread acceptance of the methods, definitions, and findings of recent psychiatric epidemiological studies, the perennial problems of the discipline remain unresolved. Although the absence of a precise psychiatric nosology before 1980 was a major barrier to epidemiological inquiry, the uncritical application of the DSM categories through the use of the DIS and CIDI conceals a number of continuing problems. Psychiatric epidemiologists have long recognized that the principal problem in the field is how to define a psychiatric case (Plunkett and Gordon 1960). This requires having valid criteria that can distinguish cases of disorder from cases of nondisorder. Psychiatric epidemiologists assume that both the DIS and CIDI are valid indicators because of their generally high correlations with clinical interviews. However, both the survey instruments and the interviews rely on the same DSM criteria, which themselves have questionable validity (e.g., Horwitz and Wakefield 2007; Kirk and Kutchins 1992). Indeed, it is far more difficult to construct conceptually valid criteria of disorder in epidemiological studies than in clinical samples.

Contemporary psychiatric epidemiologists assume that their establishment of standardized case-finding techniques, development of uniform and consistent criteria, and application of reliable differential diagnoses has resolved the problem of distinguishing the presence of valid mental disorders in community populations from the distress created by problems of living. This assumption is grounded in the belief that responses to survey questions about the presence of symptoms in epidemiological studies are identical to responses to similar questions asked in clinical interviews. Yet, the diagnoses that stem from descriptive, a-contextual instruments such as the DIS and CIDI in epidemiological studies have two major differences from the diagnoses of patients in clinical samples. First, patients typically enter treatment only after making contextually influenced self-diagnoses that their symptoms are not simply responses to stressful circumstances, which will disappear when the stressor dissipates or will gradually go away over time (Horwitz and Wakefield 2006). Second, clinicians also make contextual judgments that their patients’ symptoms do not solely reflect distressing social contexts. Thus, in treated populations, contextual judgments of both patients and clinicians precede clinical diagnosis.

In contrast, the diagnostic process in epidemiological surveys, which involve neither self-evaluations by respondents nor clinical judgments, ignores the context in which symptoms develop and thus cannot demonstrate the validity of its case-finding techniques. It cannot separate symptoms that arise because of realistic situations, personality dispositions, or physical conditions, on the one hand, or mental disorders, on the other. Psychiatrists John Wing, J. Cooper, and Norman Sartorius (1974, 135) explained this well in regard to an anxious patient: “In clinical terms, an individual may worry because he has something to worry about, because he is a worrier, because he has phobias, because he has depressive preoccupations, (or) because he has persecutory delusions.” Epidemiological studies, however, cannot make these basic distinctions and treat all symptoms as possible signs of a mental disorder.

Survey interviewers are forbidden to discuss with respondents the intent of the survey questions, exercise clinical discretion, or use flexible probes of responses. For example, in the ECA study, the most commonly reported symptoms of depression are “trouble falling asleep, staying asleep, or waking up early” (33.7 percent); being “tired out all the time” (22.8 percent); and “thought a lot about death” (22.6 percent) (Judd et al. 1994). College students during exam periods and people who must work overtime, who are worried about an important upcoming event, or who take the survey soon after the death of a famous person all would naturally experience such symptoms, which neither respondents nor clinicians would see as requiring treatment but which may nevertheless qualify as signs of disorder in community surveys. Moreover, common diagnoses such as depression require only two weeks of duration. This means that epidemiological studies count as disorders many transitory and self-correcting states that would rarely enter clinical treatment.

Epidemiological studies magnify the flaws of the symptom-based categories in the diagnostic manuals since the DSM-III because they treat all symptoms, regardless of the context in which they developed, as potential signs of mental disorders. In clinical venues, entry into treatment presumes that the symptoms are not simply responses to stressful circumstances. But in community-based samples, symptoms often are a natural result of social stressors. The most prevalent diagnoses in community surveys—mood and anxiety disorders—lack even minimally adequate criteria to separate proportionate responses to serious losses and threats from mental disorders. The use of symptom-based measures is, therefore, far more problematic in epidemiological than in clinical settings. The failure of psychiatric epidemiology to develop adequate measures of context and thus conceptually valid criteria continues to bedevil attempts to provide accurate estimates of rates of mental illness in untreated populations.

The use of a-contextual measures that consider all persons who report enough symptoms as having a mental disorder generates the huge estimates of the prevalence of mental disorders in community populations just cited. Such estimates are valuable for psychiatry and other mental health professionals and policymakers because they create the impression that mental disorder is a “public health problem” of immense magnitude. As Robert Spitzer shrewdly noted,

Researchers always give maximal prevalence for the disorders that they have a particular interest in. In other words, if you're really interested in panic disorder, you're going to say it's very common. You never hear an expert say, “My disorder is very rare.” Never. They always tend to see it as more common. (quoted in Lane 2007, 77)

Spitzer's point holds with even greater force when the overall amount of mental illnesses is considered: the enormous number of putative mental disorders that epidemiological studies uncover serves to elevate the status and importance of the psychiatric profession, psychiatric epidemiologists, and mental health policymakers because of the magnitude and severity of the problems that they confront. Moreover, the enormous rates of psychiatric disorder that epidemiological studies reveal can deflect attention away from the smaller number of persons who suffer from serious, debilitating, and chronic conditions and who ought to be the focus of policy efforts.

However useful such large and increasing estimates might be for enhancing the legitimacy and importance of psychiatrists and other mental health professionals and policymaking bodies, they result from methods that treat symptoms of mental illness and of distressing, but normal, emotions alike as signs of psychiatric disorder. The problems of establishing the validity of the descriptive measures of symptoms that psychiatric epidemiological studies use have not been solved: they have only been swept under the rug.

Conclusion

The history of psychiatric epidemiology in the United States has had a changing relationship with public policy. The field emerged in the latter part of the nineteenth century as a product of efforts to use institutional data to enhance the functioning of mental hospitals. By 1920 its focus had turned to exploring the relationship between the social environment and rates of mental illness, under the assumption that mental disorders resulted from socioeconomic conditions. This assumption coincided with the general theoretical thrust of the psychiatric profession during this period. Since the 1970s, psychiatric epidemiology has become tightly intertwined with the psychiatric profession. Just as the previous generation of psychiatric epidemiological studies reflected the psychosocial emphasis that dominated psychodynamic psychiatry, current studies embrace the individualistic framework the profession adapted in the 1970s. Reflecting the new emphasis on intra-individual factors, policy implications of epidemiological studies have now turned from a focus on changing the social environment toward identifying and treating distressed individuals at the earliest possible age (Insel and Fenton 2005).

Instead of reflecting growing progress as a scientific enterprise, the history of American psychiatric epidemiology to this point demonstrates the power of professional interests and ideology to shape definitions of what counts as a case of mental illness, assumptions about the causes of mental illness, and policies regarding persons with mental illness. Psychiatric epidemiology's perennial problem of developing valid criteria to accurately measure the prevalence of mental illnesses in untreated community populations remains unresolved, as does its ultimate goal of developing causal relationships between the environment and psychiatric disorders. Until it develops a valid nosology that can separate mental disorders from natural distress, the field is unlikely to realize these important goals.