Abstract
The US Food and Drug Administration's Sentinel System was established in 2009 to use routinely collected electronic health data for improving the national capability to assess post‐market medical product safety. Over more than a decade, Sentinel has become an integral part of FDA's surveillance capabilities and has been used to conduct analyses that have contributed to regulatory decisions. FDA's role in the COVID‐19 pandemic response has necessitated an expansion and enhancement of Sentinel. Here we describe how the Sentinel System has supported FDA's response to the COVID‐19 pandemic. We highlight new capabilities developed, key data generated to date, and lessons learned, particularly with respect to working with inpatient electronic health record data.
Early in the pandemic, Sentinel developed a multi‐pronged approach to support FDA's anticipated data and analytic needs. It incorporated new data sources, created a rapidly refreshed database, developed protocols to assess the natural history of COVID‐19, validated a diagnosis‐code based algorithm for identifying patients with COVID‐19 in administrative claims data, and coordinated with other national and international initiatives.
Sentinel is poised to answer important questions about the natural history of COVID‐19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for COVID‐19 prevention and treatment.
Keywords: COVID‐19, electronic health records, medical claims, real‐time monitoring, real‐world data, surveillance
Key Points.
The US FDA enhanced the Sentinel System to assess the use and performance of drugs and biologics in preventing and treating COVID‐19.
A natural history master protocol to guide COVID‐19 studies that rely on real‐world data was developed, as was a protocol to study arterial and venous thrombotic events among patients with COVID‐19.
Monitoring systems have been established that are sustainable and flexible, including routine reports examining inpatient characteristics, treatments, and outcomes of COVID‐19.
Many of these activities are expected to be useful for other surveillance and epidemiology initiatives.
Sentinel's diverse network of partners, varied sources of electronic healthcare data, and deep experience of maintaining such as system has enabled Sentinel to support FDA's public health emergency response.
1. INTRODUCTION
The urgency and intensity of the public health response to the COVID‐19 pandemic has underscored the need for widespread, near real‐time, and efficient surveillance to characterize large and diverse patient populations and monitor medical product use, safety, and effectiveness. The US Food and Drug Administration (FDA)'s roles in pandemic response necessitated an expansion and enhancement of one of the Agency's core active surveillance capabilities, the Sentinel System. Sentinel has implemented an array of pandemic response activities, building on more than a decade of partnerships and growing analytic capacity. Here we describe how the Sentinel System has supported FDA's response to the COVID‐19 pandemic, demonstrating its ability to generate important, timely data and descriptive epidemiology for ongoing decision‐making and to inform and enable future treatment studies.
1.1. Overview of the Sentinel System
The Sentinel System was established in 2009 to use electronic health data for improving the national capability to assess post‐market medical product safety, with an intent to also serve a broader set of needs for FDA and others. 1 , 2 , 3 , 4 Sentinel's data are based primarily on administrative claims and electronic health records (EHRs). The core Data Partner organizations include national and regional health plans partners, integrated delivery systems, and Medicare‐fee‐for service data. Sentinel partnerships also include EHR‐based organizations and networks—HCA Healthcare, the National Patient‐Centered Clinical Research Network (PCORnet), and several EHR data aggregators (TriNetX, Veradigm, IBM Explorys)—who contribute data on a case‐by‐case basis.
The claims data available from Sentinel's core Data Partners encompass approximately 800 million person‐years of longitudinal data and accrue information on approximately 70 million individuals per year, with representation from all 50 states. Most Data Partners can obtain full‐text medical records for the purpose of validating their electronic data or providing clinical detail. Claims data have the advantage of assuring nearly complete ascertainment of medically attended events during defined periods for each person, a feature lacking in most US‐based EHR data. In contrast, EHR data have more detail about care provided within the organizations that maintain them, and inpatient EHRs are particularly useful to assess hospital inpatient care, for which there is limited detail in claims data.
Sentinel operates as a privacy‐preserving distributed data network, 5 , 6 with each organization maintaining possession and operational control over its data. To allow efficient and consistent implementation of a study across the System's distinct sources, each core Data Partner transforms their data into the Sentinel Common Data Model, allowing identical computer programs to be run across locations. The Common Data Model includes demographic information, ambulatory and inpatient diagnoses and procedures, outpatient drug dispensings, laboratory test results, vital signs, and dates of eligibility for care. 7 Each organization's data are reviewed each time it is refreshed and only approved for analysis once it has passed extensive quality assurance checks. For most analyses, executable computer programs are securely distributed to each partner for execution against the curated data, with results returned securely for aggregation and analysis. FDA has used Sentinel to conduct hundreds of privacy‐preserving analyses that have contributed to drug safety communications and other regulatory decisions. 8 Nearly 200 peer‐reviewed publications have been produced. 9
The Sentinel projects described below are public health surveillance activities conducted under the authority of the FDA and, accordingly, are not subject to Institutional Review Board oversight (Basic HHS Policy for Protection of Human Research Subjects, 45 CFR §46.102(l)(2)). 10
1.2. Medical countermeasure activities in the pre‐COVID‐19 era
Several key Sentinel initiatives completed before the pandemic laid the groundwork for COVID‐19 activities. In 2018, the FDA began leveraging advances in real‐world evidence generation by building capacity to monitor treatments and outcomes during a public health crisis without burdening the medical system. 11 As part of that initiative, we enhanced Sentinel to collect and analyze data on treatments and outcomes relevant to a public health emergency. 12 One aspect of this medical countermeasure related work focused on HCA Healthcare's inpatient EHR data from 140 affiliated hospitals in 18 states. Designed to allow near real‐time monitoring of influenza—a proxy for a public health emergency—the system was reconfigured to assess COVID‐19 in April 2020. As described below, Sentinel can actively monitor the characteristics, treatments, and outcomes of patients with COVID‐19 in these data. Other prior work in influenza, including use of near real‐time claims data and novel methods for active safety surveillance, contributed to Sentinel's ability to rapidly respond to COVID‐19. 13 , 14 , 15
2. COVID‐19 ACTIVITIES WITHIN THE SENTINEL SYSTEM
Early in the pandemic Sentinel developed a multi‐pronged approach to expand its capabilities to support FDA's anticipated data and analytic needs. FDA's interests in using Sentinel data include the assessment of drug and biologic use in the prevention and treatment of COVID‐19 and preparation for assessing the performance of such products. We enhanced the data model, incorporated new data sources, and created a rapidly refreshed database. Sentinel also (a) developed new protocols, including a “Master Protocol” to assess the natural history of COVID‐19 as the pandemic continued, providing a foundation for evaluating new therapies, (b) undertook a study to validate COVID‐19 diagnosis in claims data, and (c) coordinated with other national and international initiatives. As part of FDA‐Catalyst, 16 which supplements Sentinel with access to patient reported data, FDA modified its MyStudies smartphone app to enable virtual participant consent in trials. Taken together, these projects support the Agency's ability to study COVID‐19 disease and its treatments.
2.1. Data model and database enhancements
We added SARS‐CoV‐2 laboratory test results to the Common Data Model. To provide timely analyses, we worked with several large national health plans and regional integrated delivery systems to increase the frequency and reduce the latency for updates of the claims portions of the distributed data (typically 4–6 months from time of care), refreshing core elements of the distributed database and COVID‐19 diagnostic test results every approximately 4–6 weeks.
2.2. Validation of COVID‐19 diagnostic codes
The most specific way to define a COVID‐19 cohort for analysis is to include only patients with a positive diagnostic test for SARS‐CoV‐2. Currently, the Centers for Disease Control and Prevention considers a positive nucleic acid amplification detection test as confirmatory. 17 However, most claims data sources, including Medicare, have incomplete laboratory test results, yet comprehensively capture other medical care. Therefore, it is essential to develop diagnosis code‐based algorithms which accurately identify COVID‐19 to augment laboratory‐confirmed cases to identify COVID‐19 cohorts. We conducted a validation study in six health plans of multiple algorithms based on International Classification of Diseases, Tenth Revision, Clinical Modification diagnosis codes for identifying individuals hospitalized with COVID‐19. 18 We found that the COVID‐19‐ specific code U07.1, available since April 2020, had a positive predictive value of 81.2% (95% CI, 80.1%–82.2%) and sensitivity of 94.9% (95% CI, 94.2%–95.5%). The high performance of the algorithm provides confidence for using diagnosis codes to identify hospitalized COVID‐19 individuals in the absence of complete laboratory data for many epidemiologic purposes.
2.3. Natural History of COVID‐19—Master protocols and routine analyses
2.3.1. Natural history master protocol
Because COVID‐19 is a novel disease, its natural history must be documented to provide essential reference data for new treatments. We developed a natural history Master Protocol to serve as a guide for an array of COVID‐19 studies that rely on claims, EHR, or linked claims and EHR data. This work enables studies funded by FDA and others to share common approaches and definitions where appropriate. Data obtained from this protocol will allow researchers to address the feasibility of planned future studies, and to identify limitations and potential biases. Computable phenotypes for defining cohorts, pre‐existing conditions, COVID‐19 disease severity, and complications are defined based on epidemiologic and clinical expertise. The protocol includes code lists for data elements. Special populations, including pregnant women and children, are addressed. This Master Protocol is available on the Sentinel website. 19
2.3.2. Thrombotic events among people with COVID‐19
Numerous reports have described thrombotic events among patients hospitalized with COVID‐19. 20 , 21 , 22 To guide the study of these events, we developed a protocol to estimate the incidence of arterial and venous thrombotic events among patients with COVID‐19. It compares the risk of these events among those with COVID‐19 to patients with seasonal influenza using propensity score‐based adjustment. This will allow us to elucidate whether COVID‐19 infection increases the risk of thrombosis compared to another common respiratory virus with pandemic potential. The protocol also addresses the identification of risk factors for arterial and venous thrombotic events, particularly patient characteristics that promote stasis of circulation (e.g., obesity, atrial fibrillation), endothelial injury (e.g., diabetes, hypertension), and hypercoagulability (e.g., cancer, history of prior venous thromboembolism).
Within Sentinel, the protocol is being implemented with data from integrated delivery systems and national claims partners, building on the strengths inherent to each source. This protocol has been posted on the Sentinel website 23 and was disseminated through the COVID‐19‐Evidence Accelerator sponsored by the Reagan‐Udall Foundation, 24 guiding work by others.
2.3.3. Natural history of COVID‐19 in the inpatient setting
To monitor patterns of care and clinical outcomes, we leveraged Sentinel's pandemic preparation activities to develop a surveillance report of COVID‐19 census and clinical characteristics in HCA Healthcare. We identified hospitalizations with a COVID‐19‐specific diagnosis code, as well as those with a positive SARS‐CoV‐2 diagnostic test performed within the system, generating routine reports examining patient demographic and medical characteristics that may increase a person's risk of complications. We identified medication administration and assessed complications during the hospitalization. We used standardized nursing documentation data to capture parameters of oxygen‐related therapy; a clinician reviewed the information and mapped the entries to a set of specific oxygen‐related therapies. We developed markers of illness severity as well as ordinal endpoints (a categorical disease severity scale), which are commonly used in clinical trials. The data are updated every 2 weeks and analyses are restricted to discharged patients with complete billing, an important consideration as otherwise data are incomplete, a challenge for use by public health agencies. 25
Evaluating HCA Healthcare data between February 20, 2020, and January 10, 2021, we identified 96 414 hospitalizations with a COVID‐19 diagnosis among 1 404 596 total hospitalizations. Table 1 shows the characteristics of hospitalized patients diagnosed with COVID‐19 stratified by treatment with select medications. The following diagnoses were commonly documented during hospitalization among the COVID‐19 patients: hypertension (62%), diabetes (43%), hematologic disorders (38%), liver or renal disorders (38%). Approximately 3% had evidence of pregnancy. The most common agents used to treat hospitalized COVID‐19 patients included azithromycin (58%), dexamethasone (62%), remdesivir (33%), and remdesivir plus dexamethasone (31%). Most patients had low‐molecular‐weight heparin administrations (65%). We developed an ordinal endpoint assessing mechanical ventilation, intensive care stay, extracorporeal membrane oxygenation, and death. Overall, the median length of stay was 6 days, 47% were treated in an intensive care unit, 75%were mechanically ventilated or on supplemental oxygen, and 11% died in hospital. Figure 1 shows the proportion of those diagnosed with COVID‐19 who received selected medications. There was an increase in remdesivir following FDA emergency use authorization 26 and an increase in dexamethasone following publication of the RECOVERY trial. 27
TABLE 1.
Characteristics of hospitalized patients with COVID‐19 diagnoses, February 20, 2020‐January 10, 2021, HCA Healthcare Sentinel System data
| COVID‐19 hospitalizations (N = 96 414) | COVID‐19 hospitalizations with dexamethasone administration (N = 60 188) | COVID‐19 hospitalizations with remdesivir administration (N = 32 050) | COVID‐19 hospitalizations with heparin administration (N = 23 933) | COVID‐19 hospitalizations with low molecular weight heparin administration (N = 62 934) | |
|---|---|---|---|---|---|
| Total patients associated with COVID‐19 hospitalizations | 89 127 | 58 247 | 31 898 | 23 278 | 60 972 |
| Age at admission | |||||
| 18–29 yrs | 4588 (5%) | 1807 (3%) | 847 (3%) | 564 (2%) | 2502 (4%) |
| 30–59 yrs | 33 327 (35%) | 21 135 (35%) | 11 580 (36%) | 7052 (29%) | 24 349 (39%) |
| 60–69 yrs | 19 529 (20%) | 13 088 (22%) | 7336 (23%) | 5308 (22%) | 13 373 (21%) |
| 70–79 yrs | 20 219 (21%) | 13 228 (22%) | 7148 (22%) | 5869 (25%) | 12 529 (20%) |
| ≥80 yrs | 18 751 (20%) | 10 930 (18%) | 5139 (16%) | 5140 (22%) | 10 181 (16%) |
| Sex | |||||
| Female | 46 267 (48%) | 27 622 (46%) | 14 073 (44%) | 10 367 (43%) | 29 906 (48%) |
| Race | |||||
| White | 57 153 (59%) | 36 521 (61%) | 19 935 (62%) | 13 250 (55%) | 36 744 (58%) |
| Black | 16 894 (18%) | 9761 (16%) | 4204 (13%) | 5444 (23%) | 10 391 (17%) |
| American Indian/Alaska Native | 116 (<1%) | 78 (<1%) | 41 (<1%) | 35 (<1%) | 84 (<1%) |
| Other a | 19 696 (20%) | 12 253 (20%) | 6976 (22%) | 4576 (19%) | 13 961 (22%) |
| Unknown | 2555 (3%) | 1575 (3%) | 894 (3%) | 628 (3%) | 1754 (3%) |
| Hispanic ethnicity | |||||
| Hispanic or Latino | 28 219 (29%) | 17 691 (29%) | 9774 (31%) | 6180 (26%) | 20 215 (32%) |
| Unknown | 3828 (4%) | 2449 (4%) | 1349 (4%) | 969 (4%) | 2612 (4%) |
| Length of Stay | |||||
| Mean (SD) | 8.9 (8.9) | 9.6 (8.9) | 11.6 (9.6) | 12.4 (11.9) | 9.7 (9.3) |
| Median | 6.0 | 7.0 | 8.0 | 9.0 | 7.0 |
| Documented high risk conditions | |||||
| Asthma | 6956 (7%) | 4810 (8%) | 2794 (9%) | 1500 (6%) | 5182 (8%) |
| Chronic obstructive pulmonary disease | 14 935 (16%) | 10 172 (17%) | 5531 (17%) | 4411 (18%) | 9205 (15%) |
| Diabetes | 41 385 (43%) | 27 634 (46%) | 14 893 (47%) | 13 151 (55%) | 27 021 (43%) |
| Obesity | 28 150 (29%) | 20 585 (34%) | 12 609 (39%) | 7994 (33%) | 20 614 (33%) |
| Hypertension | 60 007 (62%) | 39 470 (66%) | 20 646 (64%) | 18 036 (75%) | 39 127 (62%) |
| Atrial fibrillation | 13 374 (14%) | 8675 (14%) | 4665 (15%) | 4596 (19%) | 6728 (11%) |
| Ischemic heart diseases | 22 473 (23%) | 14 640 (24%) | 7269 (23%) | 8662 (36%) | 12 620 (20%) |
| Heart failure | 15 498 (16%) | 9988 (17%) | 4790 (15%) | 6182 (26%) | 7846 (13%) |
| Myocarditis | 263 (<1%) | 150 (<1%) | 88 (<1%) | 145 (1%) | 182 (<1%) |
| Pericarditis | 724 (1%) | 455 (1%) | 223 (1%) | 337 (1%) | 411 (1%) |
| Atrial arrhythmias | 13 374 (14%) | 8675 (14%) | 4665 (15%) | 4596 (19%) | 6728 (11%) |
| Ventricular arrhythmias | 2363 (3%) | 1592 (3%) | 908 (3%) | 1109 (5%) | 1440 (2%) |
| Liver and renal disorders | 36 576 (38%) | 23 539 (39%) | 11 772 (37%) | 16 666 (70%) | 20 124 (32%) |
| Hematological disorders | 37 058 (38%) | 23 389 (39%) | 12 928 (40%) | 13 200 (55%) | 25 553 (37%) |
| Smoking | 7442 (8%) | 4206 (7%) | 2100 (7%) | 1965 (8%) | 4780 (8%) |
| Malignant cancer | 3822 (4%) | 2421 (4%) | 1302 (4%) | 1243 (5%) | 2201 (4%) |
| Elixhauser comorbidity score | |||||
| Mean (median) | 6.9 (5.0) | 7.1 (5.0) | 6.7 (5.0) | 11.0 (9.0) | 6.3 (5.0) |
| Evidence of pregnancy | |||||
| Codes for pregnancy marker or gestational age | 2602 (3%) | 469 (1%) | 172 (1%) | 197 (1%) | 682 (1%) |
| Pulmonary complications on admission | |||||
| Pneumonia | 71 947 (75%) | 51 745 (86%) | 28 994 (91%) | 18 904 (79%) | 51 576 (82%) |
| Acute respiratory failure | 49 734 (52%) | 38 956 (65%) | 24 520 (77%) | 13 366 (55.8%) | 37 198 (59%) |
| Chronic respiratory failure | 515 (1%) | 302 (1%) | 122 (<1%) | 138 (1%) | 260 (<1%) |
| Acute bronchitis | 1375 (1%) | 875 (2%) | 442 (1%) | 251 (1%) | 1007 (2%) |
| Lower respiratory infection | 140 (<1%) | 8 (<1%) | 3 (<1%) | 32 (<1%) | 53 (<1%) |
| Acute respiratory distress syndrome | 2000 (2%) | 1439 (2%) | 1102 (3%) | 990 (4%) | 1465 (2%) |
| Thromboembolic outcomes on admission | |||||
| Pulmonary embolism | 2016 (2%) | 1378 (2%) | 767 (2%) | 1015 (4%) | 1296 (2%) |
| Deep venous thrombosis | 1384 (1%) | 883 (2%) | 536 (2%) | 787 (3%) | 862 (1%) |
| Disseminated intravascular coagulation | 298 (<1%) | 209 (<1%) | 130 (<1%) | 149 (1%) | 176 (<1%) |
| Myocardial infarction | 5019 (5%) | 3363 (6%) | 1667 (5%) | 2938 (12%) | 2737 (4%) |
| Unstable angina | 626 (1%) | 296 (1%) | 139 (0%) | 308 (1%) | 334 (1%) |
| Ischemic limb | 22 (<1%) | 10 (<1%) | 3 (<1%) | 11 (<1%) | 12 (<1%) |
| Ischemic stroke | 1735 (2%) | 847 (1%) | 387 (1%) | 682 (3%) | 836 (1%) |
| Hemorrhagic stroke | 424 (<1%) | 213 (<1%) | 91 (<1%) | 160 (1%) | 137 (<1%) |
| Other complications on admission | |||||
| Sepsis | 31 654 (33%) | 22 515 (37%) | 13 709 (43%) | 10 084 (42%) | 22 601 (36%) |
| Oxygen delivery b | |||||
| Bilevel positive airway pressure (BiPAP) | 12 487 (13%) | 10 507 (18%) | 7774 (24%) | 5251 (22%) | 9445 (15%) |
| High flow | 21 700 (23%) | 17 721 (29%) | 13 700 (43%) | 7459 (31%) | 17 016 (27%) |
| Nasal cannula | 66 708 (69%) | 48 762 (81%) | 28 631 (89%) | 17 466 (73%) | 47 917 (76%) |
| Nonrebreather | 14 529 (15%) | 11 223 (19%) | 8133 (25%) | 5240 (22%) | 11 081 (18%) |
| Oxygen conserving device | 3553 (4%) | 2908 (5%) | 2053 (6%) | 1106 (5%) | 2888 (5%) |
| Simple mask | 11 113 (12%) | 8162 (14%) | 5746 (18%) | 4075 (17%) | 8211 (13%) |
| Mechanical ventilation during stay | 15 488 (16%) | 12 132 (20%) | 8554 (27%) | 7209 (30%) | 10 941 (17%) |
| Extracorporeal membrane oxygenation (ECMO) | 44 (<1%) | 29 (<1%) | 24 (<1%) | 27 (<1%) | 33 (<1%) |
| Intensive care unit (ICU) stay | |||||
| Any ICU stay | 44 955 (47%) | 29 695 (49%) | 17 979 (56%) | 14 500 (61%) | 30 333 (48%) |
| Discharge status | |||||
| Home | 61 051 (63%) | 39 708 (66%) | 20 891 (65%) | 11 805 (49%) | 42 918 (68%) |
| Discharged to skilled nursing facility, intermediate care facility, or other | 24 507 (25%) | 14 112 (23%) | 7153 (22%) | 7748 (32%) | 14 460 (23%) |
| Expired | 10 856 (11%) | 6368 (11%) | 4006 (13%) | 4380 (18%) | 5556 (9%) |
| Ordinal endpoints | |||||
| Death | 10 856 (11%) | 6368 (10%) | 4006 (13%) | 4380 (18%) | 5556 (9%) |
| ICU, mechanical ventilation, or ECMO | 9145 (10%) | 7264 (12%) | 5265 (16%) | 3888 (16%) | 6789 (11%) |
| ICU, no mechanical ventilation or ECMO | 28 250 (29%) | 17 839 (30%) | 9720 (30%) | 6909 (29%) | 19 460 (31%) |
| No ICU stay | 48 163 (50%) | 28 717 (48%) | 13 059 (41%) | 8756 (37%) | 31 129 (49%) |
| Select medications administered | |||||
| Azithromycin | 55 800 (58%) | 40 675 (68%) | 23 028 (72%) | 14 131 (59%) | 41 413 (66%) |
| Hydroxychloroquine | 4980 (5%) | 961 (2%) | 454 (1%) | 1703 (7%) | 3434 (6%) |
| Hydroxychloroquine and azithromycin | 4126 (4%) | 684 (1%) | 329 (1%) | 1444 (6%) | 2853 (5%) |
| Losartan | 8527 (9%) | 5865 (10%) | 3353 (11%) | 2115 (9%) | 5996 (10%) |
| Remdesivir | 32 050 (33%) | 29 393 (49%) | 32 050 (100%) | 7929 (33%) | 26 469 (42%) |
| Dexamethasone | 60 188 (62%) | 60 188 (100%) | 29 393 (92%) | 15 708 (66%) | 44 788 (71%) |
| Remdesivir and Dexamethasone | 29 393 (31%) | 29 393 (49%) | 29 393 (92%) | 7138 (30%) | 24 337 (39%) |
| Tocilizumab | 2206 (2%) | 1398 (2%) | 1061 (3%) | 907 (4%) | 1 921 (3%) |
| Methylprednisolone | 14 900 (17%) | 8428 (14%) | 6774 (21%) | 4633 (19%) | 11 504 (18%) |
| Norepinephrine | 9659 (10%) | 6941 (12%) | 4715 (15%) | 5644 (24%) | 6570 (10%) |
| Heparin | 23 933 (25%) | 15 708 (26%) | 7929 (25%) | 23 933 (100%) | 9442 (15%) |
| Low‐molecular‐weight heparin | 62 934 (65%) | 44 788 (74%) | 26 469 (83%) | 9442 (40%) | 62 934 (100%) |
| Anti‐platelet therapy | 8188 (9%) | 4025 (7%) | 1944 (6%) | 2494 (10%) | 4870 (8%) |
| Positive COVID‐19 diagnostic test c | |||||
| Record of any inpatient test (PCR, antigen, or unknown test type) | 70 931 (79%) | 43 716 (79%) | 22 998 (79%) | 18 243 (82%) | 47 614 (80%) |
| PCR positive among those with an inpatient PCR test | 55 175 (94%) | 33 076 (95%) | 17 057 (95%) | 13 765 (94%) | 37 769 (95%) |
Note: COVID‐19 hospitalizations are those with one or more of the following International Classification of Diseases, Tenth Revision, Clinical Modification diagnosis codes: B97.29 (Other coronavirus as the cause of diseases classified elsewhere), U07.1 (COVID‐19), or B34.2 (Coronavirus infection, unspecified). Patients are those who have been discharged, with complete billing. HCA Healthcare has reported on other information identified in their data regarding COVID‐19. 40 , 41 Cohorts categorized by medical product administration are not mutually exclusive. Intensive care unit stays were identified by revenue codes.
“Other” race includes Asian patients.
Oxygen use is derived from oxygen‐related nursing documentation.
This reflects laboratory data available through December 31, 2020; the overall number of COVID‐19 hospitalizations through December 2020 was 89 928. Only inpatient SARS‐CoV‐2 tests performed within an HCA Healthcare facility are available.
FIGURE 1.
Proportion of COVID‐19 hospitalizations with administration of select medications, by week, February 20, 2020‐January 10, 2021, HCA Healthcare Sentinel System data. The gray bars represent COVID‐19 hospitalizations independent of medication administrations. The US Food and Drug Administration issued an emergency use authorization (EUA) for remdesivir on May 1, 2020. 26 Preliminary results from the RECOVERY trial on dexamethasone were released June 16, 2020 27 [Colour figure can be viewed at wileyonlinelibrary.com]
Several monoclonal antibodies have been authorized for emergency use against COVID‐19 by FDA. Sentinel plans on using HCA and the rapidly refreshed database to assess their use and safety by monitoring for infusion‐related reactions and other potential safety issues, should they arise.
2.3.4. COVID‐19 in pregnancy
In order to study the impact of COVID‐19 on pregnancy, as large a sample of pregnant women with COVID‐19 as possible is needed. Toward that end, the FDA has partnered with the European Medicines Agency (EMA). The EMA is initiating a project called CONSIGN (“COVID‐19 infectiOn aNd medicineS In preGNancy”) that will study the natural history of COVID‐19 disease in pregnant women across eight European countries. 28 Sentinel has joined this international collaboration, implementing a study aligned with CONSIGN. We will use Sentinel's validated analytic tools to estimate the prevalence of select medicines in pregnant women with and without COVID‐19, and in nonpregnant women with COVID‐19. We will also describe disease severity and clinical outcomes of pregnant women with COVID‐19, according to treatments, age, and trimester during pregnancy, compared to nonpregnant women with COVID‐19.
2.4. Monitoring of critical drugs—inpatient and outpatient
There is potential for shortages of products used to treat COVID‐19 patients, as well as other drugs, due to supply chain interruptions. Sentinel developed new capabilities early in the pandemic to assess regional and temporal patterns of inpatient use of 120 critical medications. With a need for near real‐time monitoring and geographic diversity we leveraged partnerships with several database systems including HCA Healthcare and multiple EHR data aggregators including TriNetX, Veradigm, and IBM Explorys. Each partner has access to near real‐time EHR data that includes prescriptions and medication administrations. Inpatient medication utilization is reported along with measures of COVID‐19 burden, defined as rates of hospitalizations by state and week. We routinized reporting for these metrics, with historic data for comparison. Similar to Figure 1, we have observed variation and changes in medication utilization that coincide with medical knowledge advances or policy events. We have also observed regional variation in the use of some drugs.
2.5. COVID‐19 MYSTUDIES app for e‐consent
As part of the broader Sentinel Initiative, FDA developed the MyStudies smartphone app in 2018 to enable the collection of patient‐reported data to augment routine Sentinel data. 29 To respond to the pandemic, the app was modified to facilitate volunteers' enrollment in clinical trials, providing a way for participants to securely, electronically consent to a trial when in‐person contact is not possible or practical. The changes enable electronic consent for any eligible trial, not just those for COVID‐19 related products. The COVID MyStudies app is currently available in the Apple App and Google Play Stores and 11 studies are currently available for electronic consent. A Spanish language version is underway.
2.6. Dissemination and collaboration
From its inception, Sentinel has been committed to transparency. From code lists to analytic programs, we have made public the details of analyses and have maintained this commitment during the pandemic. The master protocols described above are seen as public resources, facilitating work within and outside of FDA. Sentinel investigators regularly present during meetings of the Reagan‐Udall Foundation's COVID‐19 Evidence Accelerator where we have discussed specific projects as well as broader topics such as approaches to data quality management and how to define COVID‐19 cohorts. 30 , 31 , 32 , 33 , 34 , 35 Through a collaboration organized by the International Coalition of Medicines Regulatory Authorities on COVID‐19 Real‐World Evidence and Observational studies, FDA committed to collaborating with the EMA on the CONSIGN study of COVID‐19 in pregnancy. 36 FDA is also collaborating with the National Heart Lung and Blood Institute (NHLBI) to describe coagulopathy in real‐world settings, to support NHLBI's ACTIV trials. 37 The EMA will conduct a study aligned with the one described above examining thrombotic events among patients with COVID‐19 and a meta‐analysis may ultimately be conducted with the EMA‐ and FDA‐sponsored results. 36
2.7. Lessons Learned
The pandemic has underscored the importance of timely nation‐scale clinical information. While EHRs are now routinely used, the ideal data—EHR information from every provider and facility, linkable with a unique patient identifier to create a complete and clinically rich longitudinal patient record—are unavailable. Inpatient EHR data provide important details but working with such data requires a deep understanding of the underlying source data, workflows, and documentation practices to comprehensively identify, for example, medications of interest within heterogenous systems. Further, even with direct access to such data, there can be challenges with identifying certain types of care such as oxygen use. Mechanical ventilation or supplemental oxygen use, indicators of disease severity, are likely underestimated if only procedure codes are relied upon, 38 necessitating supplemental access to nursing documentation, diagnosis codes, or other data sources for more complete ascertainment. Indeed, when we used nursing documentation information in HCA Healthcare, we identified substantially more patients with indicators of severe disease. In initial analyses we found that 75% of patients diagnosed with COVID‐19 had evidence of oxygen use or mechanical ventilation, compared to 26% when procedure codes were used.
3. CONCLUSIONS
The nation requires public health surveillance systems capable of monitoring the occurrence of serious illnesses as well as assessing the use, safety, and effectiveness of treatments. FDA's long‐term investment in Sentinel has made possible the program's ability to respond to the COVID‐19 pandemic. We have initiated a wide range of projects using data from many sources. The products of these initiatives are expected to be useful beyond FDA's needs. Numerous monitoring systems have been established that are sustainable and flexible. Sentinel is poised to answer important questions about the natural history of COVID‐19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for prevention and treatment. Routinely collected electronic health information, coupled with commitment to a network of diverse partners, can fulfill the long‐held promise of supporting essential public health needs during an emergency. 39 This ability depends on FDA's sustained investment in understanding dynamic local variations in documentation, coding, and storage of data, the appropriate use of different data for different purposes, and expertise in the scientific and practical aspects of maintaining a system such as Sentinel.
CONFLICT OF INTEREST
The authors declare no conflict of interest.
ACKNOWLEDGMENTS
The authors thank the following individuals and organizations for their contributions to this work: Gregory Measer; Austin Cosgrove; Edward Rosen; Katherine Haffenreffer; Maria Kempner; Soowoo Back; Casie Horgan; Robert Jin; Christine Draper; Jennifer Lyons; Mayura Shinde; Ashley Martinez; Laura Hou; Jenice Ko; Jillian Burk; Jennifer Noble; Meg Her; Adee Kennedy; Juliane Reynolds; Max Ehrmann; Bridget Nolan; Amanda Maynard; Christopher Martin; Jamie Nolan; Alexander Peters; Samuel Attaya; Neha Varma; Abinav Gowda; Joy Kolonoski; Julie E. Richard; Laura Shockro; Tawil Contreras; Christine Lee Halbig; Claudia Coronel‐Moreno; Neesha Nathwani; Linda Curtis; Alexander Mai; Daniel Scarnecchia; Sampada Nandyala; Justin Vigeant, Suzanne Carter; Jolene Damon; Jolene Mosley; Monisha Billings; Natasha Pratt; Marie C. Bradley; Allyson Pishko; Dena M. Carbonari; Sean Hennessy; Laura McLean; Michael Klompas; Aileen Ochoa; TriNetX; Veradigm; IBM Watson Health; Aetna, a CVS Health company; Duke University School of Medicine, Department of Population Health Sciences, Durham, NC, through the Centers for Medicare and Medicaid Services; Harvard Pilgrim Health Care Institute; HealthCore, Inc. (Anthem, Inc.); HealthPartners Institute; Humana Healthcare Research Inc.; Kaiser Permanente Colorado Institute for Health Research; Kaiser Permanente Hawaii Center for Integrated Health Care Research; Kaiser Permanente Mid‐Atlantic States, Mid‐Atlantic Permanente Research Institute; Kaiser Permanente Northwest Center for Health Research; Kaiser Permanente Washington Health Research Institute; Marshfield Clinic Research Institute; OptumInsight Life Sciences Inc.; Vanderbilt University Medical Center, Department of Health Policy, through the TennCare Division of the Tennessee Department of Finance & Administration.
The Sentinel System is sponsored by the US Food and Drug Administration (FDA) to monitor the safety of FDA‐regulated medical products. The Sentinel Operations Center is funded by the FDA through the Department of Health and Human Services Contract Number 75F40119D10037. This work was supported in part by HCA Healthcare. The views expressed in this publication represent those of the authors and do not necessarily represent the official views of HCA Healthcare or any of its affiliated entities.
Cocoros NM, Fuller CC, Adimadhyam S, et al. A COVID‐19‐ready public health surveillance system: The Food and Drug Administration's Sentinel System. Pharmacoepidemiol Drug Saf. 2021;30:827–837. 10.1002/pds.5240
Members of the FDA‐Sentinel COVID‐19 Working Group: Catherine Corey, MSPH; Grace Chai, PharmD; Sarah K. Dutcher, PhD; Wei Hua, MD; Brian Kit, MD; Silvia Perez‐Vilar, PhD; Danijela Stojanovic, PhD; Corinne Woods, MPH.
Prior postings and presentation: The protocols described in the text have been posted to the Sentinel System website. One of the projects summarized was presented at ICPE 2020: Validation of Claims‐based Algorithms to Identify Hospitalized COVID‐19 Events within the FDA Sentinel System.
Funding information HCA Healthcare; US Food and Drug Administration (Department of Health and Human Services Contract Number, Grant/Award Number: 75F40119D10037)
Contributor Information
Noelle M. Cocoros, Email: noelle_cocoros@harvardpilgrim.org.
And the FDA‐Sentinel COVID‐19 Working Group:
Catherine Corey, Grace Chai, Sarah K. Dutcher, Wei Hua, Brian Kit, Silvia Perez‐Vilar, Danijela Stojanovic, and Corinne Woods
REFERENCES
- 1. Behrman RE, Benner JS, Brown JS, McClellan M, Woodcock J, Platt R. Developing the Sentinel System—a national resource for evidence development. N Engl J Med. 2011;364(6):498‐499. [DOI] [PubMed] [Google Scholar]
- 2. Robb MA, Racoosin JA, Sherman RE, et al. The US Food and Drug Administration's sentinel initiative: expanding the horizons of medical product safety. Pharmacoepidemiol Drug Saf. 2012;21(Suppl 1):9‐11. [DOI] [PubMed] [Google Scholar]
- 3. Ball R, Robb M, Anderson SA, Dal PG. The FDA's sentinel initiative—a comprehensive approach to medical product surveillance. Clin Pharmacol Ther. 2016;99(3):265‐268. [DOI] [PubMed] [Google Scholar]
- 4. Platt R, Brown JS, Robb M, et al. The FDA sentinel initiative—an evolving National Resource. N Engl J Med. 2018;379(22):2091‐2093. [DOI] [PubMed] [Google Scholar]
- 5. Maro JC. Design of a National Distributed Health Data Network. Ann Intern Med. 2009;151(5):341‐344. [DOI] [PubMed] [Google Scholar]
- 6. Brown JS, Holmes JH, Shah K, Hall K, Lazarus R, Platt R. Distributed health data networks: a practical and preferred approach to multi‐institutional evaluations of comparative effectiveness, safety, and quality of care. Med Care. 2010;48:S45‐S51. [DOI] [PubMed] [Google Scholar]
- 7. Curtis LH, Weiner MG, Boudreau DM, et al. Design considerations, architecture, and use of the Mini‐Sentinel distributed data system: Use of the mini‐sentinel distributed database. Pharmacoepidemiol Drug Saf. 2012;21:23‐31. [DOI] [PubMed] [Google Scholar]
- 8. Completed ARIA Assessments & Impact | Sentinel Initiative [Internet]. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/assessments/aria-overview/completed-aria-assessments-impact
- 9. Publications & Presentations | Sentinel Initiative [Internet]. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations
- 10. Electronic Code of Federal Regulations (eCFR) [Internet]. Electronic Code of Federal Regulations (eCFR). [cited 2021 Jan 26]. https://www.ecfr.gov/
- 11. Measer GT, Maher CT, Hu‐Primmer J. Monitoring and assessment of medical countermeasures as part of a public health emergency response. Am J Public Health. 2018;108(S3):S224‐S226. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Assessing Sentinel System Capability to Collect and Analyze Medical Countermeasure Data for the FDA Office of Counterterrorism and Emerging Threats (OCET) | Sentinel Initiative [Internet]. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/methods-data-tools/methods/assessing-sentinel-system-capability-collect-and-analyze-medical
- 13. Yih WK, Kulldorff M, Sandhu SK, et al. Prospective influenza vaccine safety surveillance using fresh data in the Sentinel System. Pharmacoepidemiol Drug Saf. 2016;25(5):481‐492. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Yih WK, Maro JC, Nguyen M, et al. Assessment of quadrivalent human papillomavirus vaccine safety using the self‐controlled tree‐temporal scan statistic signal‐detection method in the sentinel system. Am J Epidemiol. 2018;187(6):1269‐1276. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Cocoros NM, Panucci G, Haug N, Maher C, Reichman M, Toh S. Outpatient influenza antivirals in a distributed data network for influenza surveillance. Influenza Respir Viruses. 2018;12(6):804‐807. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. FDA‐Catalyst Projects [Internet]. Sentinel Initiative. 2016. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/methods-data-tools/fda-catalyst-projects
- 17. Coronavirus Disease 2019 (COVID‐19) | 2020 Interim Case Definition, Approved August 5, 2020 [Internet]. [cited 2020 Dec 23]. http://nndss/conditions/coronavirus-disease-2019-covid-19/case-definition/2020/08/05/
- 18. 2020 ICPE Presentation: Validation of Claims‐based Algorithms to Identify Hospitalized COVID‐19 Events within the FDA Sentinel System [Internet]. Sentinel Initiative. 2020. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations/2020-icpe-presentation-validation-claims-based-algorithms
- 19. Master Protocol Development: COVID‐19 Natural History [Internet]. Sentinel Initiative. 2020. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/methods-data-tools/methods/master-protocol-development-covid-19-natural-history
- 20. Klok FA, Kruip MJHA, van der Meer NJM, et al. Incidence of thrombotic complications in critically ill ICU patients with COVID‐19. Thromb Res. 2020;191:145‐147. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Lodigiani C, Iapichino G, Carenzo L, et al. Venous and arterial thromboembolic complications in COVID‐19 patients admitted to an academic hospital in Milan. Italy Thromb Res. 2020;191:9‐14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Llitjos J, Leclerc M, Chochois C, et al. High incidence of venous thromboembolic events in anticoagulated severe COVID‐19 patients. J Thromb Haemost. 2020;18(7):1743‐1746. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Descriptive Assessment of the Natural History of Coagulopathy in COVID‐19 [Internet]. Sentinel Initiative. 2020. [cited 2020 Dec 23]. https://www.sentinelinitiative.org/methods-data-tools/methods/descriptive-assessment-natural-history-coagulopathy-covid-19
- 24. Evidence Accelerator Home | Evidence Accelerator [Internet]. [cited 2020 Dec 23]. https://www.evidenceaccelerator.org/
- 25. Chaves SS, Lynfield R, Lindegren ML, Bresee J, Finelli L. The US influenza hospitalization surveillance network. Emerg Infect Dis. 2015. Sep;21(9):1543‐1550. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. US FDA . Coronavirus (COVID‐19) Update: FDA Issues Emergency Use Authorization for Potential COVID‐19 Treatment [Internet]. Silver Spring, Maryland: FDA; 2020. [cited 2021 Jan 4]. https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-issues-emergency-use-authorization-potential-covid-19-treatment. [Google Scholar]
- 27. The RECOVERY Collaborative Group . Dexamethasone in hospitalized patients with Covid‐19—preliminary report. N Engl J Med. 2021; 384(8):693–704. https://pubmed.ncbi.nlm.nih.gov/32678530/. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. FRANCISCO EM . COVID‐19: EMA sets up infrastructure for real‐world monitoring of treatments and vaccines [Internet]. Amsterdam, Netherland: European Medicines Agency; 2020. [cited 2020 Dec 23]. https://www.ema.europa.eu/en/news/covid-19-ema-sets-infrastructure-real-world-monitoring-treatments-vaccines. [Google Scholar]
- 29. Wyner Z, Dublin S, Chambers C, et al. The FDA MyStudies app: a reusable platform for distributed clinical trials and real‐world evidence studies. JAMIA Open. 2020. [cited 2021 Jan 11]; ooaa0613:500‐505. 10.1093/jamiaopen/ooaa061. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Descriptive assessment of coagulopathy among COVID‐19 patients: feasibility data review [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news‐events/publications‐presentations/reagan‐udall‐foundation‐covid‐19‐evidence‐accelerator‐2. [Google Scholar]
- 31. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Assessing data source characteristics in multi‐site analyses [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations/reagan-udall-foundation-covid-19-evidence-accelerator-0. [Google Scholar]
- 32. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Defining COVID‐19 cohorts in real‐world data [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations/reagan-udall-foundation-covid-19-evidence-accelerator-1. [Google Scholar]
- 33. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Risk of thromboembolic events with COVID‐19: a sentinel system investigation – focus on endpoints [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations/reagan-udall-foundation-covid-19-evidence-accelerator-risk-0. [Google Scholar]
- 34. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Risk of thromboembolic events with COVID‐19: a sentinel system investigation – update on methods [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news‐events/publications‐presentations/reagan‐udall‐foundation‐covid‐19‐evidence‐accelerator‐risk‐1. [Google Scholar]
- 35. Reagan‐Udall Foundation COVID‐19 Evidence Accelerator . Coagulopathy assessment in patients with COVID‐19: a TriNetX analysis [internet]. Sentinel Initiative. 2020; [cited 2020 Dec 23]. https://www.sentinelinitiative.org/news-events/publications-presentations/reagan-udall-foundation-covid-19-evidence-accelerator-3. [Google Scholar]
- 36. ICMRA meeting: COVID‐19 Real‐World Evidence and Observational studies | International Coalition of Medicines Regulatory Authorities (ICMRA) [Internet]. [cited 2020 Dec 23]. http://www.icmra.info/drupal/en/covid-19/13october2020/summary
- 37. ACTIV [Internet]. Bethesda, Maryland: National Institutes of Health (NIH) [cited 2020 Dec 23]. 2020. https://www.nih.gov/research-training/medical-research-initiatives/activ. [Google Scholar]
- 38. Wunsch H, Kramer A, Gershengorn HB. Validation of intensive care and mechanical ventilation codes in Medicare data. Crit Care Med. 2017;45(7):e711‐e714. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Simonsen L, Gog JR, Olson D, Viboud C. Infectious disease surveillance in the big data era: towards faster and locally relevant systems. J Infect Dis. 2016;214(suppl 4):S380‐S385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Sands KE, Wenzel RP, McLean LE, et al. Changes in hospitalized coronavirus disease 2019 (COVID‐19) patient characteristics and resource use in a system of community hospitals in the United States. Infect Control Hosp Epidemiol. 2020;12:1‐2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Sands KE, Wenzel RP, McLean LE, et al. Patient characteristics and admitting vital signs associated with coronavirus disease 2019 (COVID‐19)‐related mortality among patients admitted with noncritical illness. Infect Control Hosp Epidemiol. 2020;15:1‐7. [DOI] [PMC free article] [PubMed] [Google Scholar]