Census Technology, Politics, and Institutional Change, 1790–2020

Abstract

This article traces the history of the methods and technology used by the Census Bureau to convert individual census responses into published tabulations. We argue that political considerations not only shaped the content and applications of the census, but also the mechanics of census taking. By focusing on federal responses to a specific technical challenge over a very long span, our narrative illuminates the long-run effects of shifting societal preoccupations on bureaucratic decision-making. The case study of the census reflects the critical and shifting role of the state and the private sector in the development of technology. For most of the twentieth century, Census Bureau administrators resisted private-sector intrusion into data capture and processing operations, but beginning in the mid-1990s, the Census Bureau increasingly turned to outside vendors from the private sector for data capture and processing. This privatization led to rapidly escalating costs, reduced productivity, near catastrophic failures of the 2000 and 2010 censuses, and high risks for the 2020 census.

A census is a political construct that reflects the ideological orientation of its creators. Legislators, intellectuals, and the public have contested the content and purposes of the United States census for 230 years. In each period, the meaning and uses of the census reflected the politics and priorities of the moment. In the 1850s, census planners suppressed information about slavery at the behest of Southern legislators; in the 1880s, the census director promoted nativist theories of race suicide; and in the 1940s census officials helped plan Japanese internment. The census is inherently political: Its original purpose was reapportionment of political representation, and in virtually every decade, winners and losers of the demographic contest have debated the legitimacy of the results. In one case—the Census of 1920—the results were ignored altogether and no reapportionment took place, as rural legislators feared losing power to the cities.¹

Political considerations not only shaped the content and applications of the census, but also the mechanics of census taking. This essay traces the history of U. S. census data capture and processing, which we define as the methods and technologies used to transform raw census responses into statistical tables. By focusing on federal responses to specific technical challenges over a very long span, our narrative illuminates the long-run effects of shifting societal preoccupations on bureaucratic decision-making. More broadly, the case study of the census reflects the critical and shifting role of the state and political forces in the development of technology.

Census technology was constrained and enabled by historical actors operating within shifting institutional structures who responded to specific political pressures and practical bottlenecks. Before 1850 census data capture and processing was decentralized, carried out by temporary piece-work employees who went door-to-door and gathering information. This system broke down with scandalous errors in 1840, leading in 1850 to radical redesign in the census form and the methods of data processing. From 1850 to 1880, the Census Office struggled to tabulate the enumeration using paper “spread-sheets” and tally marks. With increasing population, a growing number of census questions, and rising demand for detailed tabulations, by the 1880s the Census Office was responsible for the world’s largest data processing operation, and the tally-mark system was inadequate to the challenge.

Two disruptive paradigm shifts transformed the technology of census data capture. The first shift occurred in 1890, when the Census introduced punch cards and electro-mechanical tabulators. The second shift occurred seven decades later, when the Census Bureau eliminated punch cards and tabulators in favor of electronic data capture and processing. Each of these technological revolutions was triggered by Census administrators responding to bottlenecks in data capture and processing, and in both cases the major initial innovations led to decades of incremental improvements both within the Census Bureau and in the private sector.²

The history of census technology may be read as a contest between public and private actors and institutions. Census spin-offs led to the development of the two largest data processing companies of the twentieth century. For most of the twentieth century, however, Census Bureau administrators adamantly resisted private-sector intrusion into data capture and processing operations. Beginning in 1907, the Census Bureau maintained its own machine shop that designed and manufactured data processing equipment, in direct competition with machinery produced by the private sector. For nine decades the Census Bureau was able to maintain bureaucratic autonomy, doing all data capture and processing in-house, mainly using purpose-built equipment engineered and manufactured by Census Bureau staff. As Carpenter has shown, similar bureaucratic autonomy occurred across a variety of federal agencies where mid-level staff developed unique capabilities that enabled them to resist political pressures.³

Census Bureau autonomy ended abruptly in the 1990s. Ideological shifts of the late 20^th century redefined the role of government. Under pressure from the Clinton administration, the Census Bureau privatized data capture. In 1996 the Census Bureau closed the machine shop and began to outsource Census data capture operations to private vendors. Privatization led to rapidly escalating costs, reduced productivity, and near catastrophic failures of the 2000 and 2010 censuses. As we approach the 2020 Census, the risk of a major failure in data capture and processing is palpable.

Household Enumeration, 1790–1840

James Madison, the chief architect of the first U.S. Census, argued that the enumeration of the population was “an opportunity of marking the progress of society, and distinguishing the growth of every interest.” Accordingly, in January 1790 he proposed two census schedules: one for demographic basics and the other for occupations. Madison’s demographic schedule became the template for the 1790 Census, but the proposed occupational schedule was defeated amid concerns that the questions “would excite the jealousy of the people” and that the extra expense was simply “gratifying an idle curiosity.” Madison wrote to Secretary of State Thomas Jefferson, who was responsible for directing the census, “It was thrown out of the Senate as a waste of trouble and supplying materials for idle people to make a book.”⁴

The U. S. Census was the earliest regularly scheduled national-level population enumeration, and Madison’s design included a highly efficient system for data capture. There were only a few earlier censuses to serve as models, and there is no evidence that Madison knew about them.⁵ Previous censuses usually listed the name and characteristics of each individual. Madison’s census listed an entire household on each line, by recording only the name of the household head and the number of household members with each characteristic (white men 16+, white men under 16, white females, other free persons, and slaves). This layout allowed for highly efficient data capture.

There was no standard printed form in 1790. Six hundred and fifty assistant marshals gathered the information using whatever paper was available to draw forms with the column headings specified by Madison’s legislation. Although there was some variation from place to place, in general, assistant marshals followed a consistent plan, such as the page from the enumeration of Greenville, South Carolina shown in Figure 1. The marshals kept a continuous count of the number of persons with each set of characteristics, updating the totals with the completion of each page. The first row of each column, labeled “Brought Over,” or “Brought Forward,” represents the number of persons of each type recorded on prior pages. The final row of each page gives the updated total, including all the families enumerated on the page. Those numbers were then copied to the top of the subsequent page. When an assistant marshal completed his division, he was instructed to “cause a correct copy, signed by himself, of the schedule, containing the number of inhabitants of his division, to be set up at two of the most public places within the same” so that the public would have an opportunity to make corrections. Once those copies had been publicly accessible for a reasonable period, the assistants reported the totals for each category of persons to the U.S. Marshal for their district, and claimed their payment of one dollar for every 150 persons enumerated.⁶

Figure 1. 1790 Census Enumeration Page, Greenville South Carolina.

Source: First Census of the United States. National Archives and Records Administration microfilm publication M637–11, p. 87. Record Group 29. Washington,1965.

Under this system, data processing was decentralized. The assistant marshals carried out the tabulations for their divisions simply by summing each column as the pages were completed, and reporting the grand totals to the marshals. The marshals then had responsibility to “transmit to the President of the United States, the aggregate amount of each description of persons within their respective districts,” along with the returns from each assistant. Secretary of State Jefferson published the results in a fifty-six page report in 1791, virtually just as he received them from the marshals.⁷

Over the next four decades Congress experienced pressure from Jefferson, Timothy Dwight, John Quincey Adams, and many others to collect more detail about the population.⁸ By 1840 there were eighty columns of data. The information for thirty-one families was squeezed onto two sides of a printed census form that was 18.5 inches wide by sixteen inches long (Figure 2). Most columns were ⅜ of an inch wide, and the rows were ⅜ inch tall.⁹ As soon became clear, enumerators frequently made errors on the cramped form by recording information in the wrong row or column. The results were catastrophic: a post-mortem report to Congress by a committee of the American Statistical Association concluded that “it would have been far better to have had no census at all, than such a one as has been published.”¹⁰

Figure 2. 1840 Census Enumeration Page, New Bedford, Massachusetts.

Source: Sixth Census of the United States. National Archives and Records Administration microfilm publication M704–177, p. 410. Record Group 29. Washington, 1965.

The most notorious error of the 1840 census was a finding that Northern blacks had much higher rates of idiocy and insanity than those in the South. Indeed, the rate of idiocy and insanity among Northern blacks increased directly with distance from the Mason-Dixon line: in Maine, the census indicated that one in fourteen “colored” persons were insane or idiotic; in Massachusetts, it was one in forty-three, and in New Jersey, one in 297. The rates were far lower in the Southern states, ranging from one in 1,229 in Virginia down to one in 4,310 in Louisiana. This finding was widely discussed. One observer attributed the results to climate, concluding that cold weather affected the “cerebral organs of the African race.” Secretary of State John C. Calhoun wrote that the census provided “unquestionable sources” demonstrating that “in all instances in which the States have changed the former relation between the two races, the condition of the African, instead of being improved, has become worse. They have invariably sunk into vice and pauperism, accompanied by the bodily and mental inflictions incident thereto—deafness, blindness, insanity, and idiocy—to a degree without example.”¹¹

Physician and statistician Edward Jarvis uncovered the real source of the finding about race and insanity. Examining the returns closely, Jarvis found that many Northern towns reporting idiotic or insane blacks also reported a black population of zero. He concluded that assistant marshals who intended to record idiotic or insane whites sometimes inadvertently entered the information in the column intended for the “colored” population, an easy error to make on a form with eighty cramped columns. Where extremely few blacks resided, a small number of such errors had enormous consequences for the rate of idiocy and insanity among blacks. The opposite error doubtless occurred as well, but where blacks were rare and whites predominated, random mix-ups of the columns had no discernable impact on the insanity rate for whites. Such errors also had no discernable impact on blacks in the South, where both the black and the white populations were substantial. Jarvis demanded, in the name of the nation’s honor, medical science, and truth that the census must be fixed to avoid such errors in the future.

The revelation of Jarvis’ findings fueled a storm of protest. Representative and former President John Quincy Adams, citing the “atrocious misrepresentations” of the census, demanded that Calhoun, who as Secretary of State was responsible for the census, make the needed corrections. Adams wrote in his diary that Calhoun “writhed like a trodden rattlesnake on the exposure of his false report to the House that no material errors have been discovered in the census of 1840.” Nevertheless Calhoun refused to change the reported census results, calling them “unimpeachable.” Despite “the manifest and palpable, not to say gross errors” of the 1840 census, the results were allowed to stand.¹²

Census reformers were determined that future enumerations would avoid the egregious errors of the 1840 Census. For the 1850 Census, leading statisticians supported the adoption of changes that addressed the problems caused by the cramped 1840 census form. The redesign, however, completely dismantled the straightforward data capture system that Madison had developed a half-century before. As a result, the redesign created a crisis of data capture, a crisis ultimately resolved by a series of Census Office innovations that transformed data processing in the States and around the world.¹³

The Tabulation Bottleneck, 1850–1880

When Congress began to debate the 1850 Census, some legislators initially advocated a repeat of the 1840 census legislation, with merely the elimination of the contentious questions relating to disabilities. The nascent statistical community pushed back, and in March of 1849, Congress established a Census Board to determine the content of the 1850 Census. The Board recommended a system of six separate schedules to enumerate the free population, the slave population, mortality, agriculture, industry, and social statistics.¹⁴ This development not only represented a vast expansion in the scope of the census but also made obsolete the data capture method used for first six decennial censuses.

A reorganization of the 1850 form at once addressed the problems of the 1840 form and facilitated the collection of more data.¹⁵ Instead of summarizing statistics for an entire family on each line, the 1850 form listed each individual on a separate line (Figure 3). Thus, a one-person family required just one line, but a ten-person family took up ten lines. Although the number of columns on the form was reduced from eighty to just thirteen, the quantity of information collected dramatically increased. On the main schedule describing the free population, names were recorded for each family member, along with exact age in years (and in months for infants), race, sex, whether married in the past year, school attendance, literacy, value of real property, and whether deaf and dumb, blind, idiotic, insane, pauper, or convict. Occupations and birthplaces were recorded as open-ended responses, providing vastly greater detail than had been available in the broadly categorized responses to previous censuses.

Figure 3. 1850 Census Page, Rochester New York.

Source: Seventh Census of the United States. National Archives and Records Administration microfilm publication M432–531, p, 318b. Record Group 29. Washington, 1965.

The reorganization and massive expansion of the other 1850 census schedules created an unanticipated data capture crisis. Unlike the previous censuses where assistant marshals completed the tabulation in the field, the marshals were now directed to forward their completed raw schedules to the Census Office in Washington. The number of rows to be tabulated on the free population schedule rose from 3.1 million in 1840 to twenty-three million in 1850. Moreover, the new layout of the 1850 population schedule meant there were no simple columns of counts to tally; rather, each individual response had to be classified into multiple categories and then tallied. Thus, responsibility for tabulating the results shifted from the assistant marshals to Census Office clerks in Washington. Over one hundred tons of census forms had to be transformed into statistical tables for publication.¹⁶

By previous standards of post-enumeration processing conducted by the Census Office, the tabulation was a massively complex operation. To generate statistical tables, clerks made tally-marks in groups of five on pre-printed “condensing forms” for each county. There were seven different condensing forms for the population schedules, printed on sheets of usually fifteen by twenty inches, and in some cases on larger sheets. The first form, for example, included the number of free persons in five-year age groups by sex and color; number of free persons born in each state and country; number of free persons married in the past year; and the number of free persons attending school, illiterate, pauper, or convict by race and nativity. Each of the 700,000 paper enumeration forms had to be handled at least seven times, one for each condensing form. To get the job done, the Census Office hired dozens of clerks, creating a pop-up tabulating operation of unprecedented scale. By the end of 1851, the Washington office had a total of 170 staff, about 10% of the entire federal workforce in Washington and nine times the number needed to process the 1840 Census.¹⁷

The operation did not go smoothly. The massive scale of the work quickly overwhelmed the Census Office. For each of the previous censuses, the tabulated results were presented to Congress almost immediately after the returns came in, but in 1850 the new census schedules created a procedural bottleneck. In the spring and summer of 1852, oblivious to the enormity of the work before the temporary Census Office clerks, the Democratic-controlled Congress conducted a partisan investigation into the presumed inefficiency of the Whig-appointed Census Office. Predictably, Congress’s investigation did not look favorably on the Superintending Clerk of the Census and his staff.¹⁸

Given the magnitude of the task, the limited and temporary nature of the clerical staff, and the partisan charged atmosphere of the early 1850s, it was no small feat that the Census Office managed to produce: a preliminary count of the population of each state in December 1851; a 160-page “Abstract of the Seventh Census” the following year; a massive volume with over 1,000 pages of tables in 1853; and the compendium of the Seventh Census in 1854. Despite a sharp reduction in staffing by 1853, the Office soldiered on, completing a final volume on the manufacturing schedule in December 1859.¹⁹

Between the Censuses of 1850 and 1870, the tabulation bottleneck worsened. The procedure for data capture remained the same: clerks in the Census Office recorded tally marks on condensing sheets, or “spread-sheets” as they became known, and summed the tally marks to construct tables recording the number of persons in each place with each combination of characteristics. The scale of the problem, however, increased dramatically. The free population almost doubled between 1850 and 1870, partly because of the abolition of slavery in 1865. The number of questions asked on each census schedule also grew. For the population schedule, the questions asked of each individual rose from eleven in 1850 to eighteen in 1870. To meet the demands of post-enumeration data processing, the Census Office raised a “clerical force” through a system of examinations. The maximum number of clerks rose from 170 in 1850 to 438 in 1870.²⁰

In 1880, the number of questions on the population schedule rose again, from eighteen to twenty-four. More importantly, the demand for detailed statistics expanded, Congress became more willing to fund processing such statistics, and Superintendent of the Census Francis A. Walker was enthusiastic about producing them. Consequently, the number of census volumes published increased from five volumes of various sizes in 1870 to twenty-two large quarto volumes in 1880, plus a compendium. This ambitious program required a massive increase in clerical staff, which grew from a peak of 438 for the 1870 census to 1,495 for the 1880 Census. As Census Director William Rush Merriam later remarked, it had become clear “that a point must be reached, before many more decades had passed, where complete tabulation within the census period [before the next enumeration began] would be actually impossible.”²¹

Unit Record Machines, 1890–1950

In 1889, Superintendent of the Census Robert P. Porter decided that a new tabulation system was needed for the count of the 1890 census, and he organized a competition to solicit the best ideas. Three inventors responded. Each had developed competing tabulation systems based on a common idea. Charles F. Pidgin, chief clerk of the Massachusetts Bureau of Labor Statistics, developed a system using cardboard chips printed in different colors. Census information was transcribed onto the chips, using symbols to represent different characteristics. Then the chips could be sorted into piles and counted. The Pidgin system was used successfully for the 1885 Massachusetts State Census. William C. Hunt, who had worked on the 1885 Massachusetts Census, offered a simplified version of the Pidgin system, using paper slips with colored inks rather than chips. Finally, Herman Hollerith, who had worked for the Census Office in 1880, invented a machine for electric tabulation of cards using holes punched in the cards, a system he had used for tabulating mortality records in Baltimore, New York, and New Jersey in the late 1880s.²²

The three systems were all based on the concept of “unit records,” which are separate records for each case being processed. For all three systems, information on each enumerated individual was transferred from the enumeration schedule to a piece of paper or cardboard. Instead of using tally-marks on a large spreadsheet, the unit records were then counted. The initial transcription added an extra step to the process, but yielded new efficiencies whenever there was more than one table using a particular characteristic. For example, once the unit records had been sorted by race, sex, and nativity, those subgroups could be reused for multiple tabulations, with subsequent counts dividing these basic groups by detailed categories of such variables as age, birthplace, or occupation. Under the traditional tally system, each new table began from scratch, which meant duplicating the same work over and over again.

To choose between the three systems, Porter appointed a committee to conduct a contest. Each contestant was required to transfer the information on 10,491 residents of St. Louis from the 1880 census onto their chips, slips, or cards, and make a set of tables. Hollerith was the clear winner. As shown in Table 1, the punched cards were significantly faster to prepare than either the chips or slips. The real advantage of the Hollerith system, however, came in the tabulation phase: the punched cards were ten times faster than the slips, and eight times faster than the chips.

Table 1.

Data processing hours: 1889 Census Office Contest

	Transcription	Tabulation	Total
Pidgen chips	110.9	44.7	155.6
Hunt slips	144.4	55.4	199.8
Hollerith cards	72.5	5.5	77.9

The Hollerith system was faster in the tabulation phase because the counting was done electrically. Each card had room for 288 punched holes, and the position of each hole identified its meaning. The central element of the machine was the circuit-closing press, which looked a little like a waffle iron (see Figure 4). The top part of the press contained 288 spring-loaded pins corresponding to the positions of the punched holes. The bottom part had 288 small cups filled with mercury. The operator placed a punched card in the press, and pulled the handle to lower the pins. Most of the pins were pressed upwards, but wherever there was a hole in the card the pins went through and into the mercury-filled cups, creating an electrical connection.²³

Figure 4. Hollerith circuit-closing card press.

Source: H. Hollerith, US Pat. No. 395,781, “Art of Compiling Statistics,” Jan. 8, 1889.

The electrical connections activated electromagnets that advanced a set of dials. The machine could be set up to count individual holes or combinations of holes. For example, a dial might be advanced when the holes for “white,” “female,” and “native-born” were all punched. Up to forty characteristics or combinations of characteristics could be counted with each pass through the circuit-closing press. The counters could store up to 9,999 cases for each characteristic; when that limit was reached, the operator transcribed the readings from all the dials onto paper, and reset the dials to zero. In addition to the counting function, Hollerith’s machine also had a sorting box, which consisted of twenty-four compartments with spring-loaded lids. When the press was activated, an electromagnet activated the catch on one of the lids, springing the compartment open. Using the sorting box and the counters simultaneously, the cards could be sorted and counted in one operation.²⁴

Shortly after Census Day, the Hollerith system was featured on the cover of Scientific American (Figure 5.) Clerks using Hollerith machines to tabulate the 1890 census were able to process an average of 7,000 to 8,000 punched cards per day. The Census Office estimated that the system would save $580,000 in wages; Hollerith was paid $230,390 in rental fees for 56 machines for four years, or about 40% of the savings. The machines were returned to Hollerith when tabulation was completed in 1894.²⁵

Figure 5.

Scientific American, August 20, 1890

In preparation for the 1900 census, the Census Office again held a contest for tabulation systems in June 1889. This time, the only competitors were Hollerith and Pidgin. Pidgin offered three new systems: the “Automatic Mechanical Tabulation System,” the “Pin Board Electrical Tabulation System,” and the “Electrical Typewriter Tabulator.” Hollerith was so nervous about the new devices that he hired a Pinkerton detective to go up to Boston to conduct industrial espionage. He need not have worried; in the end the Hollerith machines—which were virtually unchanged from the ones used in 1890—completed the census tabulation contest in less than half the time of the best Pidgin machine. Census Director Merriam negotiated a contract with Hollerith for an annual rental of $1,000 per tabulator²⁶ In all, Hollerith supplied 311 tabulating machines, twenty automatic sorters, 1,021 punches for preparing the cards, for which he was paid $428,239 in rental fees. The tabulation was fast; the time elapsed from the census day to the publication of the population volume was the shortest since 1820.²⁷

From 1790 to 1900 the Census Office was a temporary organization. Each decade, a Census Office was established, the census was taken, and then the office was closed. This system starkly contrasted with European countries, virtually all of which had established permanent central statistical offices by the mid-nineteenth century. The inefficiency of shutting down and reopening the Census Office every decade was obvious, and most superintendents of the census since the 1850s vigorously advocated for the creation of a permanent agency. It was not until 1902 that the Census Office became permanent and soon after was renamed the “Bureau of the Census.”²⁸

The cozy relationship between the Census Bureau and the Hollerith Company ended under the new Census Director, Simon N.D. North. Hollerith had lobbied against North’s appointment, in part because North believed the rental fees for the Hollerith machines were exorbitant. The conflict led to the removal of Hollerith machines from the Census Bureau, and for a brief period the adoption of the slower Pidgin devices. North reasoned that with the establishment of a permanent Census Bureau, it made sense to have permanent equipment under Bureau control. Hollerith’s original patents were set to expire on January 8, 1906, freeing anyone to develop similar tabulators. In 1905, North successfully applied to Congress for $40,000 to develop tabulating machinery. He used these funds to establish the Census Machine Shop in 1907, enabling the Census Bureau to build and maintain its own tabulating equipment.²⁹ North wrote in his annual report, “For the small sum … the Census experts have succeeded in devising a tabulating apparatus, along lines entirely novel, which infringes no patents and which is a marvel of simplicity, of accuracy, and of rapid manipulation of punched cards. … The possibilities of saving, in the costs of compiling future censuses … are enormous.” Further, “With this machine shop, the building and repair of machinery can be done much more effectively and economically by the Bureau than by contract with private manufactures.”³⁰ The Census Bureau recruited four disaffected Hollerith engineers to carry out the work, along with James Powers, a Russian immigrant who had invented machinery for several other companies. To attract talent, the engineers were allowed to take out private patents on their discoveries, as long as their inventions remained freely available for government use.³¹

Hollerith was furious. Convinced that his patents were being infringed upon, he launched a lobbying campaign with the Secretary of Commerce and President Roosevelt. The anti-monopolist Roosevelt was unsympathetic, but when Taft became president in 1909 Hollerith managed to get North fired. The new Director, Edward Durand, nevertheless opted to continue with the technology developed in the Census Bureau Machine Shop to process the 1910 census. Less than three months before the 1910 count was to begin, Hollerith filed suit against Durand, alleging patent infringement on card-sorting machines that were being altered by the Machine Shop. Hollerith got a restraining order, halting work on the machines, but the District of Columbia court overturned the order, whereupon the lawsuit fizzled out.³²

While Hollerith’s lawsuit was still going on in 1911, James Powers resigned from his position at the Census Bureau and formed the Powers Tabulating Machine Company. Powers introduced a full line of punches, tabulators, and sorters that competed directly with the products of Hollerith’s Tabulating Machine Company. The Powers Company joined with the Remington Typewriter Company and the Rand Kardex office supply firm to form Remington Rand in 1927. Meanwhile, in 1911 Hollerith’s company merged with a commercial scale company and two makers of employee time-clocks to form the Computing-Tabulating-Recording Company (CTR), and in 1924, CTR changed its name to the International Business Machines Corporation (IBM). IBM, Remington Rand, and the Census Bureau Machine Shop dominated data processing until the late twentieth century.³³

From a twenty-first century perspective, the Census Bureau Machine Shop is extraordinary. The shop was a government manufacturing establishment, explicitly set up to compete with a private-sector vendor. The Machine Shop even recruited talent from that vendor and barely skirted (or possibly infringed) its patents. Although Hollerith’s company was not a massive trust, it did have a complete monopoly on tabulating equipment. Hollerith complained repeatedly that the government should not compete with private enterprise, but his objections fell on deaf ears. The key players in the Bureau and the executive branch believed Hollerith was holding the government hostage, since no alternative suppliers of tabulating equipment existed. Further, the Census Machine Shop had already demonstrated its ability to innovate and develop machinery that was substantially more productive than commercially available equipment.³⁴ In 1912, Director Durand pushed for increased funding for the Machine Shop, “It is desirable that an appropriation … should be made to enable the Bureau to continue the employment of as large a part of its force of patent experts, inventors, and mechanics as possible. Should the bureau lose the services of all or the greater part of its present mechanical force, it would be almost impossible to secure competent men for the expert mechanical work” which these men had “long and successful experience.”³⁵

The 1910 census was processed mainly on machines developed in the Census Machine Shop. The Powers punches frequently jammed, so the early card-punching was done on old Hollerith punching machines owned by the Census Bureau. The Machine Shop eventually reduced the problem of jamming, and about two-thirds of the 1910 cards were ultimately punched on the Powers machines. Most of the tabulation was carried out on semi-automatic tabulators designed by the Machine Shop, and the Census Bureau introduced fully automatic self-feeding tabulators after the tabulation was underway.³⁶

For the next four decades, the Census Bureau relied mainly on equipment designed and maintained by the Census Mechanical Laboratory (formerly the Machine Shop), purchasing or renting supplemental equipment when necessary. Maintaining equipment purchased from outside vendors “proved to be definitely economical,” but also “paved the way for Bureau engineers to introduce modifications in line with special needs.”³⁷

The Census Mechanical Laboratory made substantial advances in the 1940s and 50s, with the development of a multicolumn sorter and major improvements to the tabulator. Mechanical Laboratory engineer, Anthony Berlinsky, the “Thomas Edison of the Census Bureau,” was reportedly responsible for numerous mechanical improvements for data processing. In 1950, Popular Mechanics enthused that “census employees have the fanciest collection of complicated machinery the mind of many has been able to devise.” “These machines are masterpieces of ingenuity” and it was Berlinsky who was “busily at work making them more ingenious.” Berlinsky told the magazine that, “When we finally got a machine to handle 40 columns of facts or ‘holes’ in a card those guys upstairs had to go think of 20 more questions they wanted to ask on the next census so everything is obsolete. This time I’m rebuilding the multi-column sorters to combine the sorting and tabulating operations. They’ll handle 80 columns of facts. I hope it holds ‘em for a while. It probably won’t, though.” Berlinsky also “conceived the idea for constructing an automatic microfilming machine, which was designed, tested, modified as necessary, and constructed in its final form under his immediate direction.”³⁸

In the late 1940s IBM hired the Census Bureau’s Chief of Machine Tabulation, Lawrence Wilson. Soon after, IBM came out with a new unit record machine, Model 101, which combined the latest innovations in sorting and tabulation that had been developed at the Census Bureau’s Mechanical Laboratory. Described in Popular Science as a “super-dooper census gadget,” the Census Bureau leased thirty-two of these machines from IBM for the 1950 census. In an ironic turnaround of the Bureau’s appropriation of Hollerith technology in 1907, four decades later the Bureau paid the descendant of the Hollerith Company for technology originally developed by Census engineers.³⁹

Electronic Data Capture and Processing

By 1950, punch card tabulation was a mature technology. Refinements continued to be made and data processing speeds continued to improve. As the tabulation technology was perfected, the biggest bottleneck became the punch card itself. At operational peak in processing the 1950 census, the Bureau employed nearly 2,000 key punch operators for over fourteen months at a cost of nearly six million dollars to record population and housing data on punch cards.⁴⁰ The population and housing components of the 1950 census required about twenty-two gigabytes of data storage. These data were stored on 282 million eighty-column cards weighing some 600 tons. The cards were fussy and fragile, and high humidity or mishandling often made them unreadable. As processing of punched data was increasingly automated, keypunching remained highly labor-intensive. Moreover, the storage, retrieval, and organization of the massive physical collection of data began to consume a growing share of resources.⁴¹

Electronic computers offered a solution to the punch card bottleneck. The first electronic programmable computer was the Electronic Numerical Integrator And Computer (ENIAC), built from 1943 to 1946 by John W. Mauchly and J. Presper Eckert at the University of Pennsylvania Moore School of Engineering. ENIAC was a wartime project designed with the goal of calculating ballistics for artillery. Mauchly was convinced that computers would eventually have civilian applications, and he approached Morris Hansen, Statistical Assistant to the Director of the Census Bureau, to discuss Bureau needs for data processing. Beginning in 1944 and continuing over the next two years, Mauchly had multiple meetings with Hanson and other census experts “to discuss the applicability of computers to census problems.”⁴²

The University of Pennsylvania asserted intellectual property rights over ENIAC in 1946. According to Mauchly, he and Eckert were given the choice of signing over their patents to the University or resigning, and they chose the latter. Morris Hansen, meanwhile, now convinced of the potential of the electronic computer for census applications, arranged for the National Bureau of Standards (NBS) to study the feasibility of computerizing the census. Backed by Census Bureau funding, in October 1946 Eckert and Mauchly received a $75,000 study contract from NBS to draw up plans for a census computer. With preliminary funds in hand, in December 1947 the inventors established the Eckert-Mauchly Computer Corporation. The following June, the new company received the full contract worth some $300,000 to build the world’s first commercial computer for the Census Bureau. Their machine, The Universal Automatic Computer (UNIVAC) was designed for data processing. A key innovation was the use of magnetic tape for data storage, which meant that the punch cards could be discarded once their information was transferred to tape. UNIVAC was completed in March 1951, too late to contribute significantly to the 1950 census tabulation. The Bureau conducted extensive testing, however, and concluded that the machine had the potential to cut data processing costs in half.⁴³

By the time the first UNIVAC was delivered, the Eckert-Mauchly Computing Corporation had been purchased by Remington-Rand, itself a descendent of the Powers Tabulating Machine Company. Soon the UNIVAC faced competition. Until 1946, IBM’s leadership had no interest in electronic computing, but the Census Bureau contract with Eckert-Mauchly was a wake-up call. By 1949, IBM had established a substantial electronics research staff, and the company unveiled its first computer in 1953. Largely because of the strength of the IBM marketing and service divisions, IBM’s computer rentals grew rapidly By 1955 the number of installed IBM machines exceeded the number of UNIVACs.⁴⁴

Figure 7 presents a genealogy of data processing. The Hollerith Tabulating Machine Company and the Powers Tabulating Company, the two main producers of unit record machines in the early twentieth century, were both Census Bureau spin-offs started by former census employees. The Eckert-Mauchly Computer Company was established with start-up funding from the Census Bureau. After various mergers and acquisitions, Hollerith became IBM and Powers became Remington-Rand. The Eckert-Mauchly Computer Corporation was renamed Univac, and Remington-Rand purchased Univac. Thus, the two biggest computer companies of the second half of the twentieth century could both trace their roots to the census.

Figure 7.

Genealogy of Data Processing

The development of the UNIVAC did not by itself solve the punch card bottleneck. The UNIVAC made it possible to discard punch cards once the information had been transferred to magnetic tape. Initially, the only method for getting data from census form to magnetic tape was via punch cards, using a card-to-tape converting machine. Accordingly, to input the data, clerks still had had to prepare hundreds of millions of punch cards, using manual methods that had only marginally improved over the previous half century.

As soon as plans for developing the UNIVAC were confirmed, Census Bureau engineers realized its use would be limited without a new method for data input. In 1949—two years before the first UNIVAC was operational—the Census Bureau began collaborating with the National Bureau of Standards to design and build the first high-speed Optical Mark Recognition (OMR) system. The idea of making a machine to “read” marks on paper was not entirely new. In 1938, IBM came out with an electrographic test-scoring machine that used the conductivity of special graphite pencils to interpret responses on what they called “mark sense” cards and convert them into punch cards. The mark sense system was used to score SAT college admission tests, and it was adapted for data capture in the 1951 Census of Canada. The Census Bureau initially considered using the same system for the U. S., but determined it was impractical because of the larger scale of the U.S. census.⁴⁵

By 1951, the National Bureau of Standards (NBS) had settled on a purely optical sensing system that used a photoelectric cell to read marks directly onto magnetic tape. NBS engineers determined that taking negative microfilms of marked paper could maximize precision of marked paper forms, so that the blackened responses appeared as transparent spots on the negatives. They used a cathode-ray tube to send a beam of light to each position on the form in sequence; when the light penetrated the microfilm, it was sensed by a photoelectric cell, and the response was encoded on magnetic tape.⁴⁶ The basic principle was essentially the same as Hollerith’s original punch card, with an electric eye substituting for Hollerith’s mercury-filled cups. Microfilm was initially adopted as an intermediate stage because it simplified the sensing of the marks, but it immediately became apparent that film offered two practical advantages over paper. First, it was extremely compact; once filmed, the paper forms were discarded and the microfilm preserved for long-term storage. Second, reels of microfilm are much easier to handle than sheets of paper, so microfilm greatly simplified automated feeding of the machine at high speed.

The first functional Film Optical Sensing Device for Input to Computers (FOSDIC) was delivered to the Census Bureau early in 1954 (see Figure 9). The device read one page per second, and it was successfully used for digitizing several surveys. Work on an improved version designed for the 1960 census began in 1956. The NBS built a prototype, and the Census Mechanical Laboratory constructed four identical units to be used for the census.⁴⁷

Figure 9. Film Optical Sensing Device for Input to Computers (FOSDIC).

Source: U. S. Census Bureau.

Enumerators going door to door used a bubble sheet to record information in the 1960 census (Figure 10). This meant that the most labor-intensive part of data capture was carried out during the face-to-face interview. From 1850 to 1950, individual characteristics were transcribed in longhand on census forms, which were then centrally processed by hand. From 1850 to 1880, this hand processing was done by tally-marks; from 1890 to 1950, the forms were transcribed onto punch cards, so they could be tallied by machine. In a sense, the decentralization of data capture in 1960 was a throwback to the first data capture system of 1790–1840, since in both eras the front-line enumerators were the ones converting the data into a form suitable for processing.

Figure 10. FOSDIC Bubble-Sheet for 1960 Census (Detail).

Source: U.S. Bureau of the Census, United States Census of Population and Housing 1960: Principal Data-Collection Forms and Procedures (Washington, 1961), 21.

The 1960 census made unprecedented use of sampling. All households were required to respond to the “short form” questionnaire, which included just seven questions per person. At every fourth household canvassed by enumerators, a “long-form” questionnaire was dropped off, with forty-six housing questions and twenty-nine additional questions pertaining to individual characteristics. In most of the country, respondents were instructed to fill out and mail back the long-form questionnaire, which was then copied onto FOSDIC bubble sheets. The 1970 census further automated data capture. The Census Bureau constructed an address list that included most of the nation’s households, and mailed out FOSDIC forms. The mailings instructed respondents to fill in the bubbles themselves and mail back the form. Households were randomly selected to receive a short form or a long form. Over 87% of households that received a FOSDIC form filled it out and sent it back, dramatically reducing data capture costs.⁴⁸

From 1970 to 1990, Census Bureau laboratories made numerous technical advances and refinements to the data capture system. These included page-turning cameras for filming the paper forms, which one engineer described as a “Rube Goldberg device that worked in an industrial capacity in a demanding non-stop production setting,” with design features later borrowed by Kodak. FOSDIC was redesigned and computerized, and the 1990 device was twenty-two times faster than the machines used to process the 1960 census. Census engineers built 21 FOSDIC machines for 1990 and scanned a billion pages of census forms in just 100 days, more than three times faster than the post-1840 record for data capture of a U.S. census.⁴⁹

Privatization, 1990–2020

President Clinton campaigned in 1992 on a “third way” platform of shrinking the federal government; he proposed to “Cut 100,000 unnecessary bureaucratic positions through attrition and mandate three percent across-the-board savings in every federal agency.” Once in office, his “reinventing government” initiative expanded this goal to a reduction of 272,900. With the support of Congress, every department in the executive branch began a program of outsourcing, and hundreds of thousands of jobs formerly performed by federal employees were transferred to private contractors. Figure 11 shows the number of executive branch employees per million Americans. The drop in federal jobs began during the Carter Administration, with almost a 10% reduction. The trend leveled off during the Reagan and first Bush administrations, but then began a precipitous drop during the Clinton years, when the ratio of federal jobs to population fell by almost 30%. Since the Clinton era, there has been little change, except for a temporary bump upwards during the Great Recession.⁵⁰

Figure 11. Executive Branch Employment per Million Population: 1975–2014.

Source: https://www.opm.gov/policy-data-oversight/data-analysis-documentation/federal-employment-reports/historical-tables/executive-branch-civilian-employment-since-1940.

In 1996, Clinton announced in his State of the Union Address that “the era of big government is over.”⁵¹ By that year, executive branch employment had already dropped by 300,000 workers. Clinton signed the Federal Activities Inventory Reform Act (FAIR) in 1998, which required federal agencies to “review their activities and define them as either inherently governmental or potentially subject to privatization,” and the number of federal employees fell by another 150,000.⁵² At the same time that the federal workforce was shrinking, the responsibilities of the federal government continued to grow. The inevitable consequence was a massive increase in the number of private contractors, who began doing work previously carried out by federal employees.

The Census Bureau was transformed by privatization. As planning for the 2000 census ramped up in the early 1990s, there was intense pressure to identify census functions that could be outsourced to private vendors. In 1993, the Clinton Administration’s privatization mandate was compounded by a hiring freeze and budget reductions, which restricted the ability of the Bureau to hire experts to investigate the potential for privatizing data capture operations. Accordingly, the Bureau outsourced the job of investigating the feasibility of outsourcing.⁵³

In February 1996, John H. Thompson, then Chief of the Decennial Management Division of the Census Bureau, wrote that based on the outsourced benefit cost analysis, “I am recommending that the Census Bureau require the use of imaging technology to perform the data capture function for the 2000 Census.”⁵⁴ The recommendation was approved, and in August 1996, the Bureau invited bids for data capture for the 2000 Census. The same year, the Bureau permanently shut down the Technical Services Division, the direct descendant of the Machine Shop established by Census Director North in 1907, signaling the end of the Census Bureau’s century-long effort to develop new technologies for data capture. Vice President Al Gore gave the Census Bureau a Hammer Award, consisting of a “$6.00 hammer with a little red, white, and blue ribbon,” which was “the Vice President’s symbolic answer to the $400.00 hammer of yesterday’s government.”⁵⁵

The decision to outsource data capture to the private sector was not a major concern for Census Bureau leadership. The Census Bureau historian conducted oral history interviews with key leaders of the era, explicitly asking them to identify the most important developments at the Bureau during their tenure; none of the leaders of the period from 1989 through 1998 mentioned privatization or outsourcing. The Clinton administration took almost two years to get a census director in place, and during that critical period an acting director authorized the initial contracts to study outsourcing of data capture. Director Martha Riche, confirmed in November 1994, had little experience in managing contracts. Michael Longini, Chief of the Census Bureau Decennial Systems and Contracts Management Office, a 25-year veteran of the Bureau, implemented privatization. Longini recruited the assistance of Commerce Department staff to identify requirements, draft requests for proposals, and evaluate bids. Despite continuing resistance from some career staff, Longini prevailed.⁵⁶

Aerospace contractors Lockheed Martin and TRW won the two major contracts for data capture in the 2000 census. Lockheed Martin won the $49 million contact for Data Capture System 2000 (DCS 2000), to provide the hardware and software needed to process paper census forms through scanning, Optical Mark Recognition (OMR), and Optical Character Recognition (OCR). OMR had been used by the Census since the introduction of FOSDIC in 1960; OCR, however, was new, and involved the automatic interpretation of open-ended handwritten census responses, reducing the need for hand-keying data. TRW won a $188 million contract for setting up and operating regional Data Capture Centers where the paper forms would be processed and converted to electronic format.⁵⁷

There were many unanticipated costs. The contractors did not consult with the Census Bureau personnel who had institutional knowledge and experience processing millions of paper forms. The Bureau’s lack of experience with contractors led to inefficiencies. Requirements were poorly documented, resulting in frequent changes. There were major philosophical differences between the contractors and the Bureau, especially in the area of quality assurance. Misunderstanding led to change orders, which increased costs.⁵⁸

The digital imaging and OCR systems did not work as well as anticipated. The scanning machines were far slower than the FOSDIC machines they replaced. It took approximately thirty-five of the new Kodak machines to do the work of a single FOSDIC machine, so the number of scanning machines grew dramatically. The 1990 census used twenty-one FOSDIC machines, which DCS 2000 replaced with 162 Kodak Digital Science Scanner 9500 machines. Despite the increased number of machines, overall data capture throughput declined by some 75%. The big advantage of the new system was its OCR capability for reading hand-written responses, but in practice the OCR proved much less reliable than had been hoped, so a high proportion of the forms had to be manually keyed.⁵⁹

As census day approached, the General Accounting Office (GAO) warned repeatedly that DCS 2000 was not ready. The system had inadequate testing and had not been demonstrated to provide the speed and accuracy needed to meet statutory deadlines for production of census results. In the 1998 dress rehearsal for the census, the data capture system crashed repeatedly “due to flaws in the system software, which could not handle the workload.” The lenses on the scanners quickly accumulated dust, so every fifteen minutes the system had to be shut down to clean them. Missing data, misinterpretation of responses, sorter jams, and a high error rate for write-in responses further plagued DSC 2000.⁶⁰

The Census Bureau had planned to use new statistical methods to improve accuracy and reduce the cost of following up with people who had not responded to the census, but in January 1999 the Supreme Court blocked the use of this approach. That meant more cost. Compounding the challenges facing the census, the technology still had serious problems. An operational test in Pomona, California, failed in November 1999. This end-to-end test of data capture revealed a much higher rate of manual keying of open-ended responses than anticipated, along with a manual keying rate only half as fast as projected. This bottleneck would slow the entire system to unacceptable levels. With six weeks to go before census day, the GAO reported “the Bureau faces formidable challenges in performing critical data capture operations.” DCS 2000 was supposed to be fully operational by October 1999; because of system requirement changes, the target date slipped to February 25, 2000, just two weeks before data capture operations were scheduled to begin.⁶¹

The operational testing of DCS 2000 showed that the system was too slow to digitize the entire census by the statutory deadline of December 31, 2000, when the statistics needed for reapportionment had to be delivered to Congress. At the eleventh hour, the Census Bureau and the contractors redesigned the workflow to meet the deadline. Instead of digitizing the entire census as it came in to the processing centers, the Bureau adopted a “two pass” approach to data capture. During the first pass the contractors processed only the items needed for the apportionment counts. Once that work was complete, contractors began work on capturing additional fields, such as income and education. This late fix required new software and hardware, as well as a substantial extension of the overall processing time.⁶²

The 2000 census narrowly averted disaster. A post-mortem study concluded that the Census Bureau had failed to capitalize on the institutional knowledge and experience of Bureau professional staff, relying instead on military contractors with no prior experience in census operations. Some census career professionals felt that continuing use of the FOSDIC system would have been far more cost-effective than outsourcing data capture. Bureau in-house content and processing experts profoundly disagreed with TRW’s decisions on data capture procedures. Subject matter experts within the Bureau complained that they could not even understand how DCS 2000 worked because the documentation was written in technical jargon. Poor understandings of data capture requirements led to frequent changes. The contracts were “cost-plus,” meaning that any increases in expenses of contractors were passed along to the Census Bureau. In the end, the cost for the Lockheed Martin contract jumped from $49 million to $220 million, and the contract to TRW went from $188 million to $314 million.⁶³

Planning for the 2010 census began before the processing of the 2000 census was complete. A key innovation for the 2010 census was elimination of the long form census questionnaire. From 1940 to 2000, the Census Bureau asked a subset of the census respondents an extra set of detailed questions. In 2000, for example, one-in-six households received a long census form with eighty-one questions. The other five-sixths of the population received a short form with just seven questions. The use of both the long form and short form reduced the burden of filling out the census form for most people, but still provided sufficient information to produce detailed statistics for individual communities. In 2010, the American Community Survey (ACS) replaced the long form. The ACS is similar to the 2000 long form, but instead of asking the questions as part of the decennial census, the ACS is spread out over the course of the decade. Each year, about three percent of the population is surveyed; by adding up the responses for several years, the ACS can provide community-level statistics similar to those from the decennial long form.

The elimination of the long form from the decennial census promised dramatic savings in data capture. In Census 2000, the long form accounted for 63% of census responses, even though it went out to only a sixth of the population. Other new technologies promised additional savings. The Census Bureau planned an internet response option, so that people could fill out the census using a web-based form. For in-person interviews of people who did not initially respond by mail, internet, or phone, the Bureau planned to use hand-held devices to capture the data at the time of the interview, along with the precise GPS coordinates of the housing unit. Both the internet response option and the handheld devices would produce tabulation-ready digital data, greatly reducing the burden of processing paper forms.

The reliance on outsourcing was even greater in 2010 than it had been in 2000. The biggest data capture contract was the Decennial Response Integration System (DRIS), which included both the paper data capture operations and the development of an internet response option. The DRIS contract went to Lockheed Martin; the initial award in 2005 was for $500 million, a dramatic jump from the size of the contracts awarded for the 2000 census. Six months later, the Census Bureau turned to another defense contractor, the Harris Corporation, awarding a $600 million contract for the Field Data Collection Automation Project. The goal of the Harris contract was development of a hand-held electronic device (Figure 12) for in-person census enumeration. Problems quickly arose. Between 2004 and 2008, the GAO issued nine warnings that the census was at risk because of mismanagement of the contractors.⁶⁴ These warnings were prescient.

Figure 12. Handheld Harris Device for 2010 Enumeration.

Source: U. S. Census Bureau

The internet response option was promptly abandoned. The 2000 Census had an internet option, but it was not widely publicized and only garnered 63,000 responses. Nevertheless, a subsequent Census Bureau evaluation judged the experiment an operational success, and made the recommendation that this response mode should be better publicized in future censuses. The Bureau conducted successful usability tests of internet response in 2003 and 2005. Despite this early progress, the DRIS contractor Lockheed Martin announced in 2006 that it could not provide an internet response facility in time for the 2008 dress rehearsal, which meant that a large-scale test would be impossible before Census day. Census Director Kincannon immediately made the surprise decision to cancel internet response, citing the risk of “phishing” and denial of service attacks, as well as the Lockheed Martin failure to deliver the system in time for large-scale testing.⁶⁵

The handheld device contract with the Harris Corporation was an even greater disaster. The software did not function correctly, the work fell behind schedule, and the projected cost more than doubled to $1.3 billion. In 2008 the Census Bureau abruptly canceled the plan to use the handheld devices for non-response follow-up and reverted to entirely paper-based processing. The last-minute change further increased the cost of the 2010 census by up to three billion dollars, making the Harris debacle one of the most expensive failed software systems in history.⁶⁶

In the wake of the cancellation of the two key innovations designed to modernize the 2010 census and reduce the cost of data capture, there was bipartisan congressional condemnation of the Census Bureau. It has been estimated that the scanning operation alone was approximately fifteen times as expensive as it would have been to type in all the 2010 Census information by hand. Senator Coburn blamed “the failure, mismanagement, and incompetence of the Census Bureau.” Senator Mikulski said the fiasco “borders on scandal,” Representative Maloney called it “a statistical Katrina,” and Representative Waxman pronounced it a “colossal failure.”⁶⁷

Other countries had greater success in modernizing their censuses. The Brazilian 2010 census was conducted entirely by in-person interviews using handheld devices for data capture. The Brazilian census had one of the longest questionnaires in the world, with thirty-seven short-form questions and 107 long-form questions, compared with just ten questions on the 2010 U. S. census. The cost for the 190,000 devices used in the Brazil enumeration was less than $100 each, a total of under $20 million. Multiple countries have used the internet to collect census data. Canada, for example, began offering an internet response option for its 2006 census, and 18% of the population took advantage of it. By the next census in 2011, 54% of Canadians responded by internet, driving the overall self-response rate up to a remarkable 85%.⁶⁸

After the cancellation of both the handheld devices and the internet response option, the 2010 U.S. census had to be conducted entirely on paper. The cost ballooned; the Lockheed-Martin contract doubled to $1.02 billion, Harris received $1.06 billion, and the other major contractors were paid $580 million. The privatization of data capture led not to the promised efficiencies, but rather to rapidly escalating costs, reduced productivity, and near catastrophic failure of both the 2000 and 2010 censuses.⁶⁹

Figure 13 describes the costs of the census from 1940 to 2010 in constant 2010 dollars. Panel A on the left shows the cost per household enumerated, since the data are collected on a household-by-household basis. By this measure, there was a 40% decline in census costs from 1940 to 1960. This savings was partly a consequence of the growing use of sampling in that era, which meant that there were fewer data items to be captured. From 1970 to 1990 census costs per household escalated rapidly, and census officials and outside critics pointed to this increase as evidence of the need for outsourcing. Privatization of census data capture in 2000, however, came with a heavy price tag, as costs per household rose by an unprecedented 79%, followed by another 30% increase in 2010.

Figure 13. Cost of Decennial Censuses per household and per item collected, 1940–2010.

Source: Steven Ruggles and Diana Magnuson, “Capturing the American People.” Minnesota Population Center Working Paper 2018–2.

Panel B on the right side of Figure 13 shows an alternative measure of census costs: the expense of the census per item of information gathered. Since 1950, costs per item have increased steadily, but the pace of increase accelerated dramatically with privatization. From 1990 to 2000, census costs per item of data collected doubled. With the elimination of the long form from the decennial census on 2010, the amount of information collected dropped in half, and the cost per item of information collected almost tripled.

The 2020 census will be almost entirely reliant on private contracts, and the scale of the work to be outsourced is greater than ever. Costs may not go up as much in 2020 as in 2000 or 2010, but there is still likely to be a significant increase. Even if the Census Bureau wanted to carry out some of the data capture operations in-house, after twenty-five years of attrition there are few employees left with the skills and expertise to carry out the work. As they did in 2000 and 2010, the GAO has designated the 2020 census as “High Risk,” raising frequent alarms about the progress of Census contractors.⁷⁰ The greatest concern for the 2020 census is the potential for information technology failure.

Census Bureau contractors are developing all-new software to replace thousands of systems developed within the Bureau over several decades. The new software is designed for all phases of data collection and processing for surveys as well as the Decennial Census. This is the largest and most complex software project ever undertaken at the Census Bureau, with total estimated costs of $5 billion, about a third of the estimated total cost of conducting the census. The largest contract, with a life cycle cost estimate of $1.28 billion, went to T-Rex Solutions LLC. T-Rex is a privately-held small business which had just $12.8 million in revenue the year prior to receiving the Census Bureau contract. To develop the census software, T-Rex is subcontracting with a dozen other vendors, including Lockheed Martin and General Dynamics. The GAO warns that IT development is behind schedule, and many critical components are incomplete. Costs have been escalating rapidly, and available funding has not been sufficient to carry out the full set of end-to-end testing that was initially planned.⁷¹

The Census Bureau is now in turmoil, and many experienced staff are departing. The Bureau is projecting large cost increases, and it is unclear whether Congress will cover them. Adding to the confusion was a transparently political attempt to insert a divisive question on citizenship, which was struck down at the last possible moment by the Supreme Court in June 2019. The 2020 census may limp across the finish line, but it is likely to be expensive and may still have high undercount of the immigrant population.⁷²

The Politics of Data Capture

Madison’s original design for the census efficiently decentralized data capture, making it feasible to gather information on millions of people with virtually no clerical staff in Washington. That system collapsed with scandalous errors in 1840, as the expanding scope of the census made Madison’s system impractical. With the reorganized individual-level census of 1850, the work of data capture shifted from enumerators in the field to the Census Office in Washington, which was soon overwhelmed by the task. The bottlenecks of centralized tallying of census forms stimulated more than a century of extraordinary technical innovation, including punch card tabulation, the first practical application of an electronic computer, and the first large-scale optical mark recognition system.

In the progressive era, the Census Bureau developed in-house engineering capabilities that directly competed with private-sector firms. Census officials felt that the Hollerith Tabulating Machine Company was making extortionate profits from their monopoly control of the tabulating business, and that a small public investment in a machine shop would be highly cost-effective. That calculation proved correct, as the Census Bureau Machine Shop produced and maintained increasingly sophisticated unit record machines for the next half century. Even after computers replaced mechanical tabulating equipment, the Census remained at the forefront of innovation in large-scale data capture by producing new technology for the electronic era. Prior to the 2000 Census, data capture operations were never outsourced, and by developing and maintaining equipment in-house the census directly competed with private companies that had been originally established as Census Bureau spin-offs.

The bureaucratic autonomy of the Census Bureau ended abruptly in the 1990s. New political pressures forced the Census Bureau to turn outward for solutions to its data capture problems rather than relying on the experience and expertise of in-house employees.⁷³ Privatization was never about saving money. Hiring defense contractors to undertake clerical work and data processing was an expensive proposition. Even the wildly optimistic cost-benefit analysis used to justify outsourcing data capture did not promise significant cost savings. The real agenda of privatization was ideological: The goal was to prove that a Democratic administration could implement Reagan’s agenda of shrinking government. Census officials portrayed data capture in the 2000 and 2010 censuses as success stories, but with respect to risks and costs the privatization project has been a dismal failure.

From the late nineteenth century to the late twentieth century, the U.S. Census was the world leader in the development and application of large-scale data capture technology. The Bureau’s research and development efforts had numerous spin-offs with transformational impact on broader society, including punch-card business accounting and commercial computing. In a broader perspective, the costs of privatizing census data capture are greater than just the increased expense of operating the census; we have also lost a valuable element of our shared institutional capital. The political tides will likely shift again, but at the Census Bureau and across the federal government it will be hard to rebuild the capabilities that the storms of privatization have swept away.

Figure 6. Census Mechanical Laboratory, 1950s.

Source: U. S. Census Bureau, https://broadcast.census.gov/pio/photos/machines_1950s_and_1960s/machines_1950_08023-hi.jpg.

Figure 8. Keypunch operator, 1950 Census.

Source: U. S. Census Bureau,