Privacy preserving record linkage for public health action: opportunities and challenges

Abstract

Objectives

To understand the landscape of privacy preserving record linkage (PPRL) applications in public health, assess estimates of PPRL accuracy and privacy, and evaluate factors for PPRL adoption.

Materials and Methods

A literature scan examined the accuracy, data privacy, and scalability of PPRL in public health. Twelve interviews with subject matter experts were conducted and coded using an inductive approach to identify factors related to PPRL adoption.

Results

PPRL has a high level of linkage quality and accuracy. PPRL linkage quality was comparable to that of clear text linkage methods (requiring direct personally identifiable information [PII]) for linkage across various settings and research questions. Accuracy of PPRL depended on several components, such as PPRL technique, and the proportion of missingness and errors in underlying data. Strategies to increase adoption include increasing understanding of PPRL, improving data owner buy-in, establishing governance structure and oversight, and developing a public health implementation strategy for PPRL.

Discussion

PPRL protects privacy by eliminating the need to share PII for linkage, but the accuracy and linkage quality depend on factors including the choice of PPRL technique and specific PII used to create encrypted identifiers. Large-scale implementations of PPRL linking millions of observations—including PCORnet, National Institutes for Health N3C, and the Centers for Disease Control and Prevention COVID-19 project have demonstrated the scalability of PPRL for public health applications.

Conclusions

Applications of PPRL in public health have demonstrated their value for the public health community. Although gaps must be addressed before wide implementation, PPRL is a promising solution to data linkage challenges faced by the public health ecosystem.

Introduction

Effective public health action depends on linking information about elements of prevention and care, including clinical interventions and outcomes, disease screening and testing, vaccine administration, and social determinants of health. The COVID-19 pandemic demonstrated how a lack of accurate, timely data can cripple efforts to proactively respond to a public health crisis.^1,2 The fragmentation of the US health care and public health ecosystem continues to pose challenges to connecting and linking data from disparate sources. Federal initiatives, such as the Centers for Disease Control and Prevention (CDC) Data Modernization Initiative and the Food and Drug Administration Enterprise Modernization Action Plan, have identified expanding data linkages as a key priority for a connected, response-ready public health system.^3,4 In addition, the Evidence Act of 2018 and the Creating Helpful Incentives to Produce Semiconductors (CHIPS) and Science Act of 2022 support the use of data for evidence-based policymaking while protecting privacy. The National Science and Technology Council and the National Institute of Standards and Technology have also focused on challenges to and mechanisms for privacy protection when linking fragmented data.⁵

Traditionally, linking data about individuals across disparate sources requires sharing personally identifiable information (PII). Using such identifiers raises privacy concerns due to the risk of identifying individuals and breaching protected health information (PHI). Federal, state, and tribal privacy laws, such as the Health Insurance Portability and Accountability Act (HIPAA), may also prohibit sharing PII and PHI. Privacy preserving record linkage (PPRL), a process for linking de-identified data, eliminates the need to share direct PII for record linkage. PPRL substantially reduces privacy and security risks and addresses a key barrier to data sharing and linkage.

While PPRL techniques have evolved substantially over the last 2 decades, bloom filter based techniques are considered as a reference standard for PPRL.⁶ Applications of PPRL are emerging across domains including health care, public health, crime and fraud detection, and national security because of its potential benefit by connecting disparate data sets and improving accuracy by de-duplicating records.^7–9 Large-scale public health initiatives have adopted PPRL to enhance collaboration across health care networks, data repositories, and research networks. For example, the National Institutes of Health (NIH) National COVID Cohort Collaborative (N3C) is piloting PPRL technology to link real-world COVID-19 data to help clinicians, researchers, and patients better understand the disease.¹⁰ CDC also used PPRL to link COVID-19 case and vaccination data and then track and analyze vaccination patterns and case counts to improve the understanding of COVID-19 vaccination effectiveness and inform public health recommendations.¹¹ The implications of data gaps that the COVID-19 pandemic revealed highlight the need to urgently explore the value of PPRL and increase knowledge about its applications, benefits, and implementation challenges. This study aims to understand the current landscape and value of PPRL in the clinical and public health domains, provide a high-level assessment of estimates of PPRL accuracy and privacy, and evaluate the opportunities and challenges in adopting and implementing PPRL for public health action.

Methods

Literature scan

The aim of the literature scan was to understand the current landscape of PPRL applications in public health from published peer-reviewed literature that was publicly available as of January 2023. The primary research question examined was whether PPRL is effective for data linkage in terms of accuracy, data privacy, and scalability. Although the review focused primarily on public health applications, articles applying PPRL to clinical care data were included because of overlapping data sources across the 2 fields. A search strategy was developed for the National Library of Medicine (NLM) National Center for Biotechnology Information (NCBI) PubMed literature database and Google Scholar. The review included articles published between January 1, 2010, and January 1, 2023. While the literature focusing on PPRL is extensive, literature about PPRL in public health is sparse. To limit the number of irrelevant articles for review, we used “OR” and “AND” operators and exact phrases, see Table 1. For example, several review articles about privacy enhancing technologies contain references to privacy preserving record linkage but are not focused on PPRL use cases or implementation. A combination of “privacy preserving record linkage” with “PPRL” or “hash” helped retrieve articles with multiple references to PPRL which improved identification of relevant articles that capture real-world use cases of PPRL. To identify state-level applications of PPRL, we also combined the search terms with individual state names.

Table 1.

Search strategy for literature scan

Database	Search strategy	Retrieved	Retained
PubMed	((((PPRL) OR (hash)) AND (privacy preserving record linkage)) NOT (plasma)) NOT (prolactin)	29	19
GoogleScholar	PPRL OR hash “privacy preserving record linkage” -plasma -prolactin OR PPRL OR hash AND “privacy preserving record linkage” AND statename -plasma -prolactin	768	11

The search resulted in an initial retrieval of 797 unique articles. Because PPRL is relatively new in the public health domain, we also searched websites of federal agencies to identify PPRL applications that may not have been published. Using the terms “PPRL” and “hash” separately on these websites did not retrieve additional unique articles or documents. Titles and abstracts were scanned to narrow the initial retrieval to a subset of publications that focused on application of PPRL methods or evaluation of PPRL effectiveness; articles that aimed to develop or refine PPRL techniques were excluded. This approach resulted in an initial subset (n = 124) of articles. Further, of these, 84 articles were subsequently excluded that did not focus on real-world applications or use cases of PPRL in public health or clinical care. An additional 13 articles were excluded that focused on evaluating the performance metrics of specific PPRL software tools. The resulting review included the remaining 30 articles. Information about public health questions answered by the study, the funding agency, datasets and number of records, type of PPRL techniques used, and key findings categorized into accuracy, privacy, and scalability were abstracted.

Key informant interviews

The literature scan focused on understanding PPRL use cases, effectiveness of PPRL to answer public health questions, and limitations of PPRL. To understand how these aspects of PPRL translate into opportunities, strategies, and challenges to PPRL adoption, implementation, and scale-up in public health, 12 interviews with subject matter experts (SMEs) from CDC were conducted. Several of the SMEs had led PPRL implementation, including for the Clinical and Community Data Initiative (CODI),¹² the COVID-19 vaccination surveillance project,¹⁰ and an evaluation of PPRL performance against clear text linkage methods.¹³ SMEs also had expertise with IT and data governance, technology, informatics, and legal services related to PPRL. We developed a semi-structured interview protocol that focused on 3 key topic areas: the value of PPRL for public health action, opportunities and strategies for PPRL adoption and scale-up, and gaps to be addressed for successful PPRL implementation in public health. The interview protocol contained questions tailored to each interviewee’s role at CDC. The interview protocol developed for members of the PPRL strategy leaders is shown in Supplementary Appendix A.

Interviews were held virtually via Microsoft Teams and generally lasted 45 min. All interviews were recorded with verbal consent from the interviewees. Two qualitative researchers transcribed the interviews for thematic analysis (LS, MB). Three researchers (LS, MB, and AP) reviewed transcripts and mapped text from the transcripts to organize the data into the 3 predefined key topic areas. For interview analysis, we followed an inductive approach to coding through iterative reading of transcripts and emergent coding. With iterative reading of the transcripts, we were able to delineate multiple interpretations of the raw data and identify cross-mapping of text to multiple topic areas. Emergent coding helped data-driven identification of key themes. Themes agreed upon by all 3 researchers to be representative of interviewee opinions were included in the final results. NVivo software was used to support data coding and analysis.

This activity was reviewed by CDC and deemed not to be research, and IRB review was not required. Activity was conducted consistent with applicable federal law and CDC policy.

Results

Literature scan

Accuracy

In real-world clinical data, PPRL has demonstrated linkage quality and accuracy comparable to clear-text linkage methods using unencrypted PII.^14–18 For example, using multiple data sources, including laboratory data, electronic health records, and payer claims in the Chicago region, data linkage using PPRL methods resulted in sensitivity and specificity as high as 100%.¹⁹ Several studies comparing PPRL linkages to a gold-standard data set—where health records belonging to the same patient were already known based on clear text linkage—have also shown that PPRL produced high accuracy, with precision and recall exceeding 90%.^20,21

Accuracy depends on several components, such as PPRL technique, and the proportion of missingness and errors in underlying data.^6,22 For example, although there was no difference in linkage accuracy when comparing PPRL to linkage with unencrypted identifiers in hospital admission records from Western Australia, the same study reported slightly lower accuracy using PPRL to link records from a different part of Australia that had a higher percentage of missing PII values.²³ Results from linking National Center for Health Statistics survey data with the National Death Index confirmed PPRL provides linkage accuracy comparable with clear text record linkage (precision ranged from 93.8% to 98.9% and recall ranged from 97.8% to 98.7%), specifically for data sets with a low proportion of missing identifiers.¹²

Data privacy

Compared with clear text record linkage, PPRL improves data privacy by design because no direct PII must be shared for linkage. Various components of PPRL implementation—including choice of PPRL method, type of PII used to generate tokens, protocols to access and store data, and participation of a linkage agent—can affect the level of privacy protection. The security of various PPRL methods has been rigorously evaluated for vulnerability to a variety of attacks and resulting modifications have led to stronger and more secure PPRL methods.^24–27 Safeguards regarding access to and the use, storage, and destruction of encrypted data are as important as the PPRL method chosen to prevent privacy breaches. For example, the HIPAA expert determination method where a privacy expert must certify that the dataset produced from the proposed linkage involves a low level of risk based on the data set, purpose of data usage, and ability of an anticipated recipient to identify an individual is a standard practice to ensure privacy and prevent re-identification.^28–30

Scalability

Recent large-scale record-linkage efforts have demonstrated the scalability of PPRL for public health. The National Patient-Centered Clinical Research Network (PCORnet) project successfully linked more than 170 million records to accurately measure clinical characteristics and disease prevalence.³¹ Similarly, NIH N3C and the CDC COVID-19 project linked millions of records from various sources and demonstrated the scalability of PPRL for public health surveillance.^9,10

Supplementary Appendix B provides a glossary of relevant terms and describes a typical workflow for PPRL implementation involving 2 data owners. Table 2 lists large-scale applications of PPRL identified from the literature scan that demonstrate the value of PPRL for public health.

Table 2.

Selected applications of PPRL in public health.

Years	Agency/organization	Description
2010*	NIH	NIH developed the Global Unique Identifier Tool, so researchers can share study participant–specific data without exposing PII, and match participants across labs and research data sets.32
2013*	Patient-Centered Outcomes Research Institute	PCORnet has de-identified and integrated the electronic health record data of over 66 million patients across the USA, which can be accessed by researchers conducting observational studies, clinical trials, population health studies, and more.31 Several research studies have utilized and expanded used of PPRL with data within the PCORnet network to answer a variety of research questions relevant to clinical care and public health.33,34
2018*	CDC	CODI linked individual-level data across 3 health systems and 3 community-based partners in Denver, Colorado, to assess the prevalence of pediatric obesity across the Metro Denver region and evaluate the effectiveness of pediatric weight management interventions and other community programs.11
2019*	CDC	The National Center for Immunization and Respiratory Diseases’ surveillance of COVID-19 vaccine administration linked vaccine administration data from 9 Immunization Information Systems data partners, 21 national and regional retail pharmacy networks, 2 dialysis partners, and 1 federal agency to track individuals’ vaccination events reported by multiple different entities (eg, vaccination clinics, pharmacies, and health care providers).10
2020*	NIH	The PPRL pilot within NIH N3C includes data from 75 institutions for over 6 million patients across the USA, for the purpose of studying the evolving coronavirus and its treatments.9
2020	US Department of Veterans Affairs (VA)	The Chicago HealthLNK Data Repository (HDR) used PPRL to link electronic health records across health care systems in the Chicago area. Data from VA were merged with data from the HDR to identify veterans eligible for VA services who were homeless or at risk of becoming homeless.16,35
2020	NIH	As part of the objective to recruit 1 million Americans in the All of Us Research Program Electronic Health Record data, the analysis aimed to determine the extent of fragmentation in care across 7 health provider organizations in 3 states (Wisconsin, Illinois, and Indiana).36
2022	National Cancer Institute	The Frederick National Laboratory for Cancer Research and Evaluation used source data from 6 US cancer registries, CVS, and Walgreens to evaluate PPRL solutions on criteria including ease of use, pre- and post-processing requirements, match quality, performance, and scalability.37
2022	CDC	Using data from Minnesota Electronic Health Record Consortium linked to data from Minnesota’s immunization information system, Homeless Management Information System, and Department of Corrections, the research team conducted a retrospective, observational cohort study evaluating COVID-19 vaccine VE against SARS-CoV-2 related hospitalization among patients who had experienced homelessness or incarceration.38
2022	CDC	The National Center for Health Statistics conducted a case study comparing initial and refined linkages of a PPRL solution to those of an established, standard method of data linkage.12

^*Indicates ongoing PPRL implementation as of January 1, 2023.

Limitations and opportunities of PPRL. PPRL is intended to protect the privacy of the individuals who are being linked. However, the literature scan also revealed weaknesses of PPRL that can impact the security and privacy of PII. The linkage quality of PPRL relies on high quality PII to construct the hash tokens, and poor data quality and linkage errors can degrade linkage quality.^16,12,23 In addition, PPRL is computationally intensive which can pose a cost and resource burden and limit the scale-up of PPRL in public health. While PPRL does not require sharing of PII, different types of adversary attacks could make PII vulnerable despite the PPRL process.³⁹ But recent developments such as BLIP (BLoom and flIP) method for bloom filter hardening (a differentially private method for flipping bits in the Bloom filter with certain probabilities) provide provable privacy protection against re-identification risks and can help mitigate these risks.^40,41

Key informant interviews

The key informant interviews revealed 4 common themes regarding gaps and strategies in PPRL adoption and scale-up:

1. Awareness and understanding of PPRL. All 12 interviewees noted that knowledge of PPRL is limited to SMEs and public health professionals who have previously implemented PPRL, and the success of PPRL depends on increasing understanding about PPRL methods, feasibility, and appropriateness for public health. All twelve interviewees noted that input from stakeholders on barriers and facilitators to PPRL implementation is important. One interviewee explained, “it took a lot [of effort] to just get to the point where there is buy-in…for funding.” Much of that effort was for education about PPRL, because there are only “a few…programs and handfuls of individuals within those programs who understand [PPRL].” Interviewees also indicated the importance of increasing awareness among public health experts and leadership in federal agencies about the benefits and limitations of PPRL, the cost and resource burden, and the return on investment for PPRL applications.

   To improve understanding of PPRL in the public health community, interviewees identified strategies such as establishing communities of practice and creating graphics and dissemination material clarifying PPRL processes and privacy preservation mechanisms. Sharing successful use cases in public health through presentations and short publications was frequently cited as a strategy for increasing trust in the benefits of PPRL across partners.

2. Data owner buy-in. Most respondents stated that buy-in and data owner participation are critical to PPRL’s success in public health. Further, all SMEs identified the engagement of state, tribal, local, and territorial public health departments (STLTs)—primary sources of public health data—was the most challenging aspect of PPRL implementation. For example, one interviewee described the difficulty engaging STLTs with the COVID-19 PPRL project was “because of the response situation and how resources are thinly stretched [during the COVID-19 pandemic], it was hard to get data owners to engage.” Multiple interviewees pointed out legal, technical, and logistical challenges that could hamper data owner consent and fidelity to PPRL protocol. All 12 interviewees highlighted the need to demonstrate the value of PPRL to data owners (eg, de-duplicating records and building community infrastructure, as demonstrated by CODI) as key to improving participation in PPRL.¹¹ Furthermore, providing education to STLT data owners about risks and benefits of PPRL and level of effort for PPRL implementation was deemed crucial to help STLTs make informed decisions regarding participating in PPRL.

3. Governance. All 12 interviewees emphasized the importance of an appropriate agency-level governance structure for PPRL, including adapting existing governance frameworks, regulations, and data-use agreements for information technology and data investments to include PPRL. Interviewees identified aspects of the PPRL process that would require developing guidance for PPRL projects, such as defining privacy and security thresholds; identifying trusted linkage agents; and creating standardized guidelines for data retention, storage, destruction, and further linkages. Ten interviewees noted that establishing processes for coordination across PPRL projects is important. Interviewees highlighted the importance of developing process checklists and guides as a way to enhance governance by outlining “at which stage do you need to know what… that road map, or an implementation path is needed [for PPRL]” and describing existing approaches and resources “a checklist for folks to go through…what data are you trying to link, usage agreements already in place, this is how our agency is approaching it, this is how we know external partners (CMS, NIH, FDA, etc.) are approaching it—we need to provide that”.

   Nine interviewees indicated that selecting a single PPRL technology or vendor within an organization or agency was paramount for establishing PPRL governance. One key informant described this as one of “the biggest [obstacles] that lay before PPRL.” Various open-source and vendor-provided PPRL solutions are available, but the use of different PPRL encryption methods and keys may preclude linking across these tools. Using a single authorized vendor or solution in an agency could overcome this issue. However, choosing a single vendor would entail time and expense, particularly if vendors or vendor ownership should change. One interviewee also noted that linking data collected by different federal agencies would require each to use the same PPRL tool, necessitating considerable coordination across agencies. Potential strategies identified to address this issue included evaluating PPRL solutions on criteria such as ease of use, pre- and post-processing requirements, match quality, license costs, performance, and scalability.

4. Collaboration within and across public health agencies. Several interviewees noted that lack of coordination and collaboration between programs implementing PPRL within public health agencies was an important barrier to its adoption and scale-up. One key informant highlighted the issues with a project-based on domain-based approach to PPRL within an agency as “if you’re just doing it in your domain, sure it'll serve one purpose. You can come out with a research paper, but has it served the bigger purpose of bringing a cohesive ecosystem, more responsive and data-driven and science-driven CDC?” Other potential issues identified included missed opportunities for data linkage, duplication of cost and effort, and the significant burden on data owners participating in multiple PPRL programs. Several interviewees indicated the need for greater collaboration across federal, state, and local agencies to share lessons learned and inform the use of PPRL for public health planning and surveillance. For example, one interviewee noted “we want to be able to find [PPRL efforts across agencies] though and learn from their experience, but there’s no way to find [information about PPRL applications from federal or state agencies] unless we dig deep…maybe a snowball type of search. We have not addressed it in a systematic way, only when we hear about it and someone’s willing to share.” Another interviewee stressed the importance of aligning with external partners to “save people time in the learning process, …. ccwe don’t want them to go down rabbit holes with vendors or inappropriate techniques.” Eight interviewees noted that providing legal, contractual, and technical guidance is important to improving PPRL adoption and scale-up.

Discussion

Although PPRL has been used for years to link disparate data across domains, knowledge of its utility in public health remains limited. Our study examined published literature and publicly available information to understand the landscape of PPRL applications and assess PPRL’s effectiveness in terms of linkage accuracy, privacy, and scalability in clinical care and public health.

Public health agencies and organizations have revealed several powerful, practical, and actionable use cases for PPRL in public health. For example, NIH N3C has used PPRL to improve research regarding the spread of and treatments for COVID-19, and the CDC COVID-19 project used PPRL to track and analyze COVID case counts and vaccination records to assess vaccine effectiveness. PPRL linkage quality was comparable to that of clear text linkage methods (requiring direct PII) for linkage across various settings and research questions.^9,10 PPRL links data in a HIPAA-compliant manner by eliminating the need to share direct PII. PPRL improves data privacy, although the level of privacy protection depends on the policies and protocols governing the storage, retention, and destruction of PPRL tokens and keys. Large-scale implementations of PPRL linking millions of observations—including PCORnet, NIH N3C, and the CDC COVID-19 project have demonstrated the scalability of PPRL for public health applications.^31,9,10

CDC subject matter experts described opportunities and challenges for adoption and scale-up of PPRL. Interviewees reported 4 main gaps and strategies in PPRL adoption and scale-up: increasing awareness and understanding of PPRL, improving data owner buy-in, establishing appropriate governance structure and policies and implementation oversight, and developing a coordinated strategy for PPRL implementation within and across public health agencies. Various agencies and organizations have recently started to develop governance frameworks for PPRL. For example, the Eunice Kennedy Shriver National Institute of Child Health and Human Development assessed potential governance and technical approaches for implementing PPRL across pediatric COVID-19 studies.⁴² PCORnet has also described the governance considerations and process to establish a standardized and scalable infrastructure for a national clinical research network.⁴³ In addition, several PPRL tools follow the secure hash standards issued by the National Institute of Standards and Technology, ensuring a robust and secure methodology for PPRL implementation.⁴⁴ Interviewees highlighted the urgent need for organizations to examine the costs, benefits, and risks including vendor lock of establishing a single approved PPRL technology solution, and the trade-offs between open-source and commercial solutions. Similar themes emerged from a CDC Foundation convening of representatives from public health organizations, industry partners, and CDC experts to discuss the potential benefits of, barriers to, and sustainable business models for PPRL implementation.⁴⁵ Interviewees suggested strategies to facilitate PPRL adoption, including demonstrating value to data owners participating in PPRL, sharing lessons learned from prior PPRL projects, and developing PPRL process checklists to help staff new to PPRL. Some of these strategies have already been applied in projects, providing insights into future PPRL applications. For example, CODI has published lessons learned from planning, designing, and implementing a clinical-community infrastructure enhancement that used PPRL technology to link data.¹¹ Similarly, the Frederick National Laboratory for Cancer Research and Evaluation used source data from 6 US cancer registries, CVS, and Walgreens to evaluate PPRL technology solutions on such criteria as ease of use, pre- and post-processing requirements, match quality, performance, and scalability.³⁷

Our study has a few limitations. The literature scan did not include articles published after January 1, 2023, and does not reflect findings from recent PPRL applications in public health. All SMEs interviewed for the study were affiliated with 1 federal agency (CDC), and therefore may not be representative of the public health community as a whole. The opportunities, successes, challenges, and limitations identified through the literature scan and SME interviews may not be fully representative of the field; as availability and adoption of PPRL evolves, it is likely that the opportunities and challenges will also evolve. Further research is needed to incorporate views of experts from other federal, state, and local agencies, health systems, researchers, and community organizations to understand barriers and facilitators to PPRL implementation across the public health community.

Conclusions

By linking patient-level clinical and public health data without sharing unencrypted PII, PPRL could bridge crucial data gaps such as lack of timely and relevant data, thus creating new opportunities for public health research and surveillance. Recent applications have demonstrated the feasibility, scalability, and value of PPRL for public health. Although gaps must be addressed before PPRL can be implemented widely across use cases, this process offers a promising solution to some of the public health ecosystem’s current data linkage challenges. Tangible next steps for improving PPRL adoption include (1) sharing lessons learned from PPRL applications, (2) checklists for those involved in PPRL selection and adoption, (3) evaluating costs, benefits, and risks of single approved PPRL technology solution, and (4) developing governance and oversight guidelines for PPRL applications.

Author contributions

The study was conceived by Adi V. Gundlapalli, and Corrine Ferdon, with Steven Luxenberg. Aditi Pathak led the design and analysis with collaboration from Daniela Zapata, Laina Serrer, and Meera Bhalla. Subject matter experts including Arunkumar Srinivasan, Raymond King, Lisa B. Mirel, Thom Sukalac, and Patrick Baier provided valuable contributions regarding real-world PPRL use cases, opportunities, challenges, and resource materials to inform the research and writing.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was funded by the Centers for Disease Control and Prevention [Contract Number HHSD2002013M53937B].

Conflicts of interest

The authors have no competing interests to declare.

Data availability

The data underlying this article cannot be shared publicly for the privacy of the individuals who participated in the key informant interviews.

Disclaimer

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the US Centers for Disease Control and Prevention and/or National Science Foundation and/or American Institutes for Research. This research was conducted while Lisa B. Mirel was employed at the National Center for Health Statistics, Centers for Disease Control and Prevention prior to becoming an employee at the National Center for Science and Engineering Statistics within the National Science Foundation.