Understanding the Role of Medical Experts during a Public Health Crisis Digital Tools and Library Resources for Research on the 1918 Spanish Influenza

Abstract

Humanities scholars, particularly historians of health and disease, can benefit from digitized library collections and tools such as topic modeling. Using a case study from the 1918 Spanish Flu epidemic, this paper explores the application of a big humanities approach to understanding the impact of a public health official on the course of the disease and the response of the public, as documented through digitized newspapers and medical periodicals.

1. Introduction

On August 17, 1918, Royal S. Copeland, Health Commissioner for New York City, responded with the following reassuring statement to reports that a ship had arrived from Europe with numerous cases of the “Spanish Flu”: “We have not felt and do not feel any anxiety about what people call ‘Spanish Influenza,’ and we considered it so unimportant that it did not seem necessary to make a public discussion of the situation.”[1] This statement, and others issued during the next several weeks that were equally reassuring and optimistic, proved to underestimate the severity of this disease, which would, by year’s end, result in the death of more than 20,000 New Yorkers due just to influenza, with many more deaths caused by related diseases such as pneumonia. During the worst period, as shown in Chart 1, the daily death rate for New York City from influenza and pneumonia doubled in the first week, and then quadrupled by the third week.

Chart 1.

New York City daily deaths due to influenza and pneumonia, as reported in Evening World, October 1918

This paper addresses a central question for any humanities scholarship: what is the role of the individual in shaping broader historical processes? In this case, how did a municipal medical official in the largest city in the United States make assessments of the danger of the influenza, issue recommendations about health policies, and more generally shape perceptions of this disease? By situating Copeland’s comments throughout the epidemic in the broader context of very large textual databases drawn from contemporary newspaper and periodical sources, we argue that a big humanities approach offers new tools to explore core questions in medical history. Scholars seeking to evaluate the significance of prominent individuals, such as Copeland, will interpret their potential influence in the context of their interactions, and exchanges with other historical figures, both individually and collectively. This approach is the historical parallel to literary close reading, as defined (and denigrated) by literary scholar Franco Moretti as “a theological exercise—very solemn treatment of very few texts taken very seriously.” As an alternative, Moretti recommends distant reading, which moves further away from a text or the texts in order to “understand the system in its entirety.” [2] This approach offers insights useful for the research question cited above by moving between layers, allowing for both close reading of significant texts and distant reading of large collections of textual evidence using automated methods such as topic modeling and tag clouds.

Our contributions in this work are:

1. We argue for a combination of close and distant reading that allows for analysis to be scaled across levels, regions, and timeframes. Medical history is a particularly suitable field for this approach to the digital humanities, because research on diseases and health necessarily transcends levels of analysis. In particular, the history of epidemics, such as the 1918 Spanish Influenza, requires analysis on every level from the microbial to the global in ways that are greatly enhanced by the capacity to read both intensively and expansively.

2. Using topic modeling algorithms we infer segmentations of the time course of news reports to characterize qualitative shifts in discourse over the course of the 1918 Spanish influenza.

3. We describe how our methodology helps better characterize the role played by Copeland in the response to the 1918 Spanish Influenza. In the case of Copeland’s reassuring prediction cited above, for example, a statement by a municipal public health official in response to a small number of cases seemingly brought from overseas would become part of the national dialogue as health officials all sought to gauge the severity of the disease outbreak.

2. Copeland’s Role: Contrasting Viewpoints

Historians engaged in traditional, close reading of Copeland’s role in guiding the public health responses to the Spanish flu in America’s largest city have produced strikingly opposite evaluations. For his critics, the reassuring statements issued at the start of the epidemic proved false and misleading in ways that potentially undermined more proactive and extensive public health measures. Historian Alfred Crosby cited an early statement by Copeland as an example of the “pollyanna attitude toward the looming pandemic” characteristic of many city health commissioners, while his declarations in December that the influenza would not return were cited as evidence that he was “still indefatigably optimistic: despite the number of cases and deaths in the intervening months.” [3] John Barry, in his popular history, The Great Influenza, was even more sharply critical of Copeland, describing him as “a man with no belief in modern scientific medicine,” who “took no action whatsoever to prevent the spread of infection” when the first cases arrived by ship in August, and that he “did nothing” after the first death occurred on September 15. It was only when the people of New York City “could see disease all about them” that Copeland took steps to enforce a quarantine on victims, according to Barry, even as he “also reassured all concerned” that the number of cases was decreasing, not increasing. [4] Charging that Copeland “continued to downplay the danger and overstate the authorities’ ability to control the disease,” historian Nancy Bristow quotes several “reassuring” statements published in newspapers in early October, to illustrate how he “continued to mouth reassurances” and “continued to deny the seriousness of the situation,” leading to this summary judgment of public health responses to the epidemic: “Though Copeland was clearly an extreme case in his efforts to reassure the public, this commitment to keeping the public calm was widespread.” Arguing that newspapers followed the Health Commissioner’s lead in attempting to “calm the public,” Bristow concludes that this effort to provide “reassurance” actually produced “indifference” which in turn led to “resistance” to “emergency measures” in many other locations. [5]

Copeland’s defenders, by contrast, seek to explain and justify his efforts in terms of the broader context of the American public health. [6] In their 2007 article published in Journal of the American Medical Association, historian Howard Markel and colleagues concluded that the New York City health department reacted “earliest to the gathering influenza crisis,” by imposing compulsory quarantine and isolation procedures, mandatory case reporting, and other public health measures, leading to the conclusion that the city’s “early and sustained response to the epidemic” contributed to the fact that the city “experienced the lowest death rate” among major East Coast cities. [7] Historian Nancy Tomes lists the significant measures, such as staggered openings for theatres and businesses, classifying influenza and pneumonia as reportable diseases, rigorous inspection of schoolchildren, enforcement of anti-spitting and anti-crowding measures in public spaces, and expanded availability of hospitals and nurses, while still avoiding the danger of “panicking a large urban population.” [8] In their study of urban school systems during the epidemic, Alexandra Minna Stern and co-authors concluded that New York City “functioned fairly smoothly during the pandemic” in part due to “Copeland’s leadership abilities,” as his “deft touch” with other officials meant that in the face of controversies and criticism, his “public demeanor always projected a sense of calm, reasonable assurance.” [9] Francesco Aimone praises Copeland’s role in shaping a “proactive approach” to the influenza epidemic through such efforts as quarantining the sick, monitoring case numbers, educating the public, and regulating personal behavior; leading to the conclusion that Copeland drew upon an appropriate combination of “regulatory and voluntary techniques” when “confronted with the overwhelming task of controlling influenza.” [10] Finally, the Influenza Encyclopedia case study of New York City describes how the Health Department quickly implemented its program of disease surveillance, isolation, and quarantine upon the arrival of the first flu cases. While citing Copeland’s reassurances that there was no cause for alarm and his “sanguine” attitude, this study identifies his simultaneous efforts to tighten disease control measures, including reporting cases, quarantining homes, public education, and expanded medical facilities. Despite the remarkable increase in the number of cases and deaths and mounting criticism of Copeland’s measures, the eventual decline in the disease and the relatively low numbers in this city led the authors to this very positive conclusion: “Through the tireless actions of Copeland and his staff at the health department, and through the amazing volunteer work of the city’s relief organizations, New York was able to weather its epidemic with a significantly lower morbidity and mortality rate than other nearby cities…Copeland could be proud of his city of the work he did [sic].” [11]

3. Exploratory Analysis

Digital collections, like Chronicling America, Hathi Trust, and Medical Heritage Library, allow for keyword proximity searches that illustrate relevance by measuring word frequency as well as data visualizations such as tag clouds, e.g., from the results of topic modeling. These methods depend on having access to the underlying API and easily downloadable text files (which is less possible with commercial vendors, such as ProQuest). The scale of these digitized collections related to a single topic in medical history can be easily illustrated through keyword searching. During the six months of the epidemic in the United States, from August 1918 to January 1919, the digitized American newspapers available from the Chronicling America collection offer 18,178 pages that include the term “influenza.” Of these results, approximately 30% (5,529 pages) appeared just in October 1918, the worst month of the epidemic. In the case of just one newspaper in New York City, the Evening World which was published six days a week, Monday to Saturday, nearly 200 pages during October 1918 included the word “influenza,” which accounted for 38.5% of all the pages published that month. On many days, including October 12, 17, 24–26, 29, and 30, as shown in Chart 2, more than one-half of pages in each issue contained the word “influenza.”

Chart 2.

Pages per issue, total and with word “influenza”

The relative frequency of the term influenza can be visualized using a tag cloud drawn from the 60,000 words that appeared in a single issue of this newspaper, October 17, 1918 (Illustration 1). After removing stop words, misspelling, and the title of the newspaper, a word cloud provides a clear and quick overview of the themes, based only on term frequency, that dominated this issue: the war (Germany, German, war, boys, Allies, privates, and army); costs and products presumably related to advertising (cents, price, savings, coats, and suits), and specifically the campaign to raise funds for the war (Liberty Bonds and Loan). Influenza appears at the top of this word cloud (marked in dark blue), but was less prominent that military and commercial terms based solely on frequency. In fact, the word “influenza” appeared 21 times in this issue, one-half as frequently as the word “army,” one-quarter as frequently as the word “Liberty,” and one-sixth as frequently as the word “German” or “Germany.” Copeland’s name appears only in one article, on page 11, where he urged landlords to provide adequate heating to ensure public health. In this same issue, Copeland also offered this laconic quote to reports that the cases had slightly decreased in two days, from 5,113 to 4,733, while deaths had increased from 317 to 336: “The figures speak for themselves.” Counting words offers one answer to the question cited above, suggesting that the influenza received less attention in this one document than other topics of the times.

Illustration 1.

Evening World, October 17, 1918

Yet keyword searching combined with close reading is a more valuable method for assessing the significance of a prominent individual. A search of two major newspapers databases for the terms “Copeland” and “influenza” produces 512 results for August to December 1918 (371 in Chronicling America and 141 in America’s Historical Newspapers). Of these results, 208 appear in three New York City newspapers and 304 results appear in national newspapers outside of New York City.

Both numbers are revealing: the former indicate the intensity of reporting within New York City on Copeland’s efforts in to control the influenza while the latter indicates how the example of this major urban center, one of the first cities to experience the influenza epidemic, and of this prominent health official, penetrated into the daily reporting of newspapers across the nation. Towards the end of the epidemic, for example, a Kentucky newspaper praised Copeland’s measures for producing a lower death rate in New York City (Illustration 2).

Illustration 2.

Daily Public Ledger, November 25, 1918

Further exploration of Copeland’s impact nationally on the spread of information about influenza can be illustrated with key word searching, which is probably the most common way for humanities scholars to use digitized materials. The first time Copeland’s name appears in any national newspaper in proximity to influenza was his recommendation, made in August 1918, that New Yorkers avoid kissing “except through a handkerchief” (Illustration 3).

Illustration 3.

Montgomery Advertiser, August 17, 1918

Identifying key terms used by Copeland makes it becomes possible to identify more examples of his influence through articles that do not use his name (and thus do not appear through the search terms cited above) yet clearly demonstrate the spread of his recommendations. Between August 17 and September 30, 1918, more than a dozen American newspapers, located through a key word search for “influenza” and “kissing,” reported on the recommendations made by Copeland, even if his name did not appear in the report, as these warnings were attributed to the New York Commissioner of Health, New York City Department of Health, or, more generally, to health authorities. [12] These recommendations against kissing as a possible means of spreading influenza were also echoed by health authorities in other locations, including the Salt Lake City Health Commissioner Samuel G. Paul, the Ohio State Department of Health, and the St. Louis Health Commissioner, Max Starkloff [13]. The capacity to do key word searching across a large number of texts is thus an important tool for digital humanists seeking new methods to understand the influence of key individuals in historical contexts.

After these August reports, more than a month passed before Copeland’s name appears again, this time in reference to the growing number of cases in New York City. Over the following weeks, however, Copeland’s specific recommendations related to public health measures were often cited, usually approvingly and at times even commercially. An advertisement by local theatre owners and operators that appeared in the Roanoke Times on October 30, 1918 (Illustration 4), for example, quoted Copeland’s statement about the importance of the theatre in times of crisis and the safety of attending a theatre that was well ventilated and clean. Including Copeland’s name in the headline is proof of his national visibility. Copeland also appeared in advertisements for chiffon veils and for soap (Illustration 5), each time with direct quotations from him emphasizing the health value of purchasing and using these products. In these cases, Copeland’s authority as both a medical expert and a public health official was invoked for commercial purposes. The capacity to do key word searching across large databases thus contributes to humanities interpretation by identifying examples of broad significance and influence.

Illustration 4.

Roanoke Times, October 30, 1918

Illustration 5.

Evening World, November 21, 1918

4. Topic modeling with segmentation

Keyword searching is an effective means to use “distant reading” techniques to identify sources that can then be subjected to “close reading.” Topic modeling serves a similar purpose of providing a quick way to assess the relative importance of terms across a very large body of text. To explore similar questions about the impact of one public health authority, Copeland, on newspaper coverage of this epidemic, we combine topic modeling with a segmentation algorithm wherein we aim to discover time points around which significant topical shift occurs. This approach was completed on three sets of data: first, all newspapers in the Chronicling America collection; second, just the New York City newspapers, and third, national newspapers excluding the New York City newspapers. This section examines reporting in all newspapers, with the goal of illustrating the breadth of coverage of this controversial figure.

Topic modeling with segmentation uses a dynamic temporal segmentation algorithm that wraps around topic modeling algorithms for the purpose of identifying change points where significant shifts in topics occur across time. The input data to the segmentation algorithm is intended to be a “bag-of-words” representation of documents, i.e., an unordered collection of words that does not take into account grammar or syntax. The main task of the segmentation algorithm is to automatically partition the total time period defined by the documents in the collection such that segment boundaries indicate important periods of temporal evolution and re-organization. For each set of topics discovered from each segment, the top 20 terms that represent the topic are tagged with their categories. The category identification is accomplished using a named entity recognition algorithm on the documents assigned to each topic. A simple word filtering technique is then applied to the discovered topics within and across segments to eliminate repeated topics and repeated terms. [14]

In the August 17–September 7 segment for all newspapers (Illustration 6), the left cloud (1) connects Copeland’s name (Royal Copeland) to his title (health commissioner and dr) and to one article in which he recommended that the spread of influenza could be prevented from using a handkerchief and refraining from kisses. The right cloud (2) refers to concerns that a ship arriving in the Atlantic port of New York City port might have brought passengers sick with Spanish flu.

Illustration 6.

All Newspapers, August 17–September 7, 1918

In the September 8–November 3 segment for all newspapers (Illustration 7), the top left cloud (1) includes references to recommended actions, including asking landlords to provide heat in apartments and assigning physicians and nurses to hospitals. The top right cloud (2) refers to warning to the public with disease terms (bacillus, epidemic, and Spanish) and public health measures (close, public, telephone, and health department). The middle cloud (3) indicates the total number of deaths reported over a period of time (yesterday, day, hour, and week) as well as relative change (increase and decrease). The bottom left cloud (4) refers to additional health measures (disease, serum and vaccine) as well as locations associated with the spread of Spanish influenza (east, street, public, city, car, and army). The bottom right cloud (5) refers to two additional public health measures: recommendations to wear a veil and the decision to close schools to prevent the spread of Spanish influenza.

Illustration 7.

All Newspapers, Sept. 8–Nov. 3, 1918

In the November 4–11 segment for all newspapers (Illustration 8), the top left cloud (1) includes family and location references (mother, father, baby, home, children, wife, and boy) that suggest the immediate effects of the influenza epidemic; the word die appears in this cloud, suggesting coverage of individual victims. The top right cloud (2) and bottom left cloud (4) suggest terms that indicate public policy and institutional responses to influenza (traffic, relief, organization, service, hospital, association, and club). The middle cloud (3) includes geographical references specific to New York City (bronx, queen, and manhattan) in connection with terms referring to changing conditions (relief/relieve). The bottom right cloud (5) has terms related to reports on health and death, including terms measuring scale of change (day, week, normal, and increase).

Illustration 8.

All Newspapers, November 4–11

Topic modeling with segmentation makes it possible to examine changes in newspaper coverage related to time and topic. Topic modeling of a medical journal, by contrast, makes it possible to situate the influence of a particular figure in the discussion of disease among medical experts. The journal, Medical Record: A Weekly Journal of Medicine and Surgery, allows for an exploration of these themes. Published in New York City, this journal included detailed reports on the 1918 Spanish Influenza and is available in digitized form for the period of the disease outbreak from both the Medical Heritage Library and the Hathi Trust. A keyword search for “influenza” in volume 94, covering the months July to December 1918, locates 229 pages that included this term, approximately one-fifth of all the pages in the journal during these six months. A search for “Copeland” produces fifteen results in this same period of time, ten of which were explicit statements about the influenza epidemic and recommended preventive measures. In the August 31, 1918 issue, for example, the journal reported Copeland’s conclusion, cited at the beginning of this paper, that the patients taken from a ship in the New York harbor were not suffering from influenza. [15] In early October, by contrast, the journal reported on the growing number of cases and public health measures, in New York City and elsewhere, that testified to the expanding danger of the epidemic. [16]. Copeland’s name does not appear on any pages in September, but coverage of the epidemic gradually increases, with a report on influenza in military training camps and an editorial stating: “There can be little doubt that it is genuine pandemic influenza…,” with case reports from European studies indicating symptoms and likely course of the disease. This issue also summarized the warning issued by U.S. Surgeon General Rupert Blue, which urged the public to seek medical attention when early symptoms of the disease were identified. [17]

Further reports on the influenza epidemic appeared in the three subsequent issues, including a detailed report by a doctor on “the origin of the so-called ‘Spanish influenza’,” which concluded with this striking recommendation: “Every patient with this epidemic should be quarantined, and everything possible should be done now at its outset to prevent its spread.” [18] The most extensive reporting on the Spanish influenza appeared in the November 2, 1918 issue, which devoted two pages to a transcript of meeting at the New York Academy of Medicine on October 17, 1918 (Illustration 9), which included “an informal discussion of influenza and of the present epidemic.” [19]

Illustration 9.

Medical Record, November 2, 1918

5. Discussion

A close reading of the contents related to influenza in this journal during the most critical stages of the epidemic suggests three broad conclusions: first, much of the information about disease in the early stages was similar to what appeared in the daily newspapers; second, sustained attention to the disease appeared only after the peak of the epidemic had been reached; and third, as the crisis worsened, the epidemic attracted increasingly “expert” attention in the form of observations of patients, records of symptoms, and specific health policy recommendations.

Similar conclusions can be reached using a very different set of methods and published materials. As described above, topic modeling allows for a distant reading of digitized textual sources. Using the same data as cited above, the journal Medical Record, topic models have been generated for the complete text, approximately 2000 pages (Illustration 10 and Illustration 10.1). This topic model shows five distinct clusters of terms: 1) Terms related to aspects of medical treatment, including wound, patients, examination, operation, and diagnosis. 2) Terms related more specifically to bodily aspects, such as blood, heart, nerve, tissue, cells, and body. 3) Terms likely related to the influenza outbreak, such as fever, treatment, pneumonia, and serum, as well as more general medical terms, not necessarily influenza-related, such as clinical, treatment, reaction, disease, diagnosis, and syphilis. 4) Terms most directly related to the war, including army, hospital, American, red cross, and department. 5) Terms related to the treatment, diagnosis, and impact of health issues, such as mental, patient, treatment, child, children, conditions, disease, and life. The term influenza does not appear in any of the other four tag clouds, suggesting that although the term appeared frequently in the last few issues of the year, the word itself was not frequently used in ways that could have clustered into a distinct topic.

Illustration 10.

Medical Record, All Text, 1918

Illustration 10.1.

Middle Segment, Medical Record, 1918

A second topic model (Illustration 11) can be created for influenza paragraphs, here defined as the three sentences immediately before and after the word influenza in the complete issues of the 1918 journal of the Medical Record. Here again, the model suggest distinct topics in the way that this journal reported in influenza in these distinct term clusters: 1) reports on influenza as a public health issue involving the military (camps and army), personnel (nurses and dr.), and facilities (hospitals); 2) reports on care, clinical study, treatment, and prevention; 3) reports on symptoms of related conditions and diseases (typhoid and pneumonia) with evaluative terms (acute and severe); 4) reports that situate the 1918 influenza in contexts defined by time (1889), location (Paris, London, and England) and medical knowledge (bacillus, plague, and epidemic); 5) reports on the influenza’s impact on specific populations (army, college, home, France, university, and hospital). Some terms appears in three or more clusters, in addition to influenza, such as medical (4), pneumonia (3), epidemic (3), and disease (3).

Illustration 11.

Medical Record, Influenza Paragraphs, 1918

As this research frames new questions about and methodologies for studying the 1918 influenza pandemic, it also reflects new and dynamic engagement with libraries and archives generally, and with historical medical libraries and archives in particular, whose collections span the centuries, open wide windows onto the human condition as it has changed over time, and are increasingly available in digital form through such initiatives as Chronicling America, Hathi Trust, and Medical Heritage Library. Herein represents an important current and future role for these institutions as they facilitate and support big humanities research, partnerships, and training for computer scientists, humanists, librarians, and other professionals. As physical and virtual worlds continue to collide and co-exist, and more individuals and teams begin (and wish to begin) to work with “big” (and inevitably bigger) data to open new avenues of research, this role for historical medical libraries and archives is today – and will be in the future – fundamentally part of, not apart from, their traditional role as stewards of original, born-physical collections whose provenance, context, and curation will remain integral to the success of any project utilizing digital surrogates and their associated data. [20]