Documentation and review of social determinants of health data in the EHR: measures and associated insights

Abstract

Objective

Electronic Health Records (EHRs) increasingly include designated fields to capture social determinants of health (SDOH). We developed measures to characterize their use, and use of other SDOH data types, to optimize SDOH data integration.

Materials and Methods

We developed 3 measures that accommodate different EHR data types on an encounter or patient-year basis. We implemented these measures—documented during encounter (DDE) captures documentation occurring during the encounter; documented by discharge (DBD) includes DDE plus documentation occurring any time prior to admission; and reviewed during encounter (RDE) captures whether anyone reviewed documented data—for the newly available structured SDOH fields and 4 other comparator SDOH data types (problem list, inpatient nursing question, social history free text, and social work notes) on a hospital encounter basis (with patient-year metrics in the Supplementary Appendix). Our sample included all patients (n = 27 127) with at least one hospitalization at UCSF Health (a large, urban, tertiary medical center) over a 1-year period.

Results

We observed substantial variation in the use of different SDOH EHR data types. Notably, social history question fields (newly added at study period start) were rarely used (DDE: 0.03% of encounters, DBD: 0.26%, RDE: 0.03%). Free-text patient social history fields had higher use (DDE: 12.1%, DBD: 49.0%, RDE: 14.4%).

Discussion

Our measures of real-world SDOH data use can guide current efforts to capture and leverage these data. For our institution, measures revealed substantial variation across data types, suggesting the need to engage in efforts such as EHR-user education and targeted workflow integration.

Conclusion

Measures revealed opportunities to optimize SDOH data documentation and review.

INTRODUCTION

A compelling body of evidence demonstrates that health and healthcare utilization are shaped by social determinants of health (SDOH), such as food security, housing stability, and transportation access.^1–3 Unmet social needs have been linked to important clinical outcomes such as uncontrolled diabetes and hypertension as well as higher levels of utilization, including hospital readmissions.^4–7 In response, prominent health organizations such as the National Academies of Sciences, Engineering, and Medicine⁸ have recommended capturing SDOH data in electronic health records (EHRs)⁹ as a key facilitator of integrating social and medical care at levels spanning research (eg, analyzing the impact of adverse childhood experiences on clinical outcomes) to care delivery (eg, connecting patients experiencing homelessness to shelters). In healthcare delivery settings, both the documentation and review of SDOH data in EHRs are critical components of integration.⁶ As a result, EHR vendors and health systems have increasingly added designated fields to promote the capture of SDOH data in structured formats. These designated fields are distinct from prior documentation approaches in which social risk factors were not differentiated from clinical needs. For example, other EHR structured fields (eg, problem lists) and unstructured fields (eg, clinical notes) can contain information about both social needs and clinical needs.^10,11

Despite growing interest in SDOH and rapid expansion of SDOH documentation tools in EHRs, there is no consensus on how different types of EHR fields should be used to capture SDOH data. As we experiment with different approaches to EHR-based SDOH documentation across health systems and care settings, measuring the impact of different approaches on achieving high levels of documentation is essential to guide optimization strategies. A small set of studies has measured the extent to which specific social risk factors are documented in EHRs in different types of fields.^11,12 For example, in a sample of patients with cardiovascular disease at a multihospital academic health system, Navathe et al¹¹ found that social risk factors such as housing instability and poor social support were more likely to be contained in physician clinical notes than in the structured problem list and health maintenance registries in the EHR. Specifically, the prevalence of poor social support increased from 0.4% when measured using structured EHR data to 16.0% when measured via clinical notes. In the community health center setting, studies by Gold et al¹³ and Cottrell et al¹⁴ measured usage of EHR-based SDOH questionnaires and found substantial heterogeneity in completion across clinic sites (4–18% in one study) with overall low uptake.

These studies reveal the value of measuring different approaches to SDOH documentation in the EHR but do not offer a generalizable set of measures that can be used to compare different EHR-based SDOH documentation options. They also do not include measures that reflect whether SDOH data are reviewed after being documented. The need for such measures is heightened as EHR vendors increasingly add SDOH fields as part of their standard products¹⁵ and health systems, in turn, more proactively plan for their uptake and use. Measuring the use of new EHR SDOH fields, and how use compares to that of preexisting fields that may contain SDOH data, can inform efforts to increase and improve EHR integration strategies.

OBJECTIVE

We sought to develop measures that would capture levels of use of the newly added structured social history fields since these fields are now widely available¹⁵ and represent a more targeted option for capturing social needs. Our measures capture 2 meaningful dimensions of use—documentation (data capture, which is conceptually akin to measuring screening) and review (data viewing). We wanted to develop measures that could also be applied to capture levels of use of preexisting approaches to SDOH documentation in the EHR to have useful comparators. Specifically, we selected 4 additional data types that featured variability along different dimensions (eg, primary role responsible for documentation, structured vs unstructured). Finally, we wanted to develop practical, scalable measures that do not require natural language processing and that can be applied in the context of a specific encounter (eg, a hospitalization) or across a patient population over a given period of time. To illustrate the types of insights that can be derived from these measures to support quality, operations, and research, we produced our measures using data from our health system.

MATERIALS AND METHODS

Setting

This study was conducted at UCSF Health, an academic medical center that includes 4 hospitals and a large number of ambulatory clinics. All UCSF sites have used an Epic EHR since June 2012. Our study sample included all patients who were hospitalized at least once at any UCSF Health hospital between February 1, 2019, and February 1, 2020. This 1-year time period was selected to reflect the 1st year in which the new structured patient social history questions were available in our EHR. During this period, we had not yet implemented Epic’s subsequently released functionality for interacting with these fields, including SDOH wheel and storyboard icons. We included the hospitalization criterion to construct encounter-level measures around hospital encounters as this encounter type represents a setting in which social needs are increasingly addressed¹⁶ and offers the best exemplar for attributing documentation and review activities (described in more detail below).

EHR documentation of SDOH: Selected data types

Patient social history questions

Potential options available for structured and unstructured SDOH documentation in EHRs have evolved over time but are largely common across modern enterprise EHR systems. For example, ICD-10 Z-codes on a problem list or clinical notes written by social workers could be documented in any EHR. Large EHR vendors, such as Epic and Cerner, have gone further and recently added structured data fields to specifically capture SDOH data.^15,17,18 The primary motivation for this study was to examine documentation and review behavior for Epic’s 2018 release that included structured fields designed to capture patients’ social needs for transportation, financial strain, and food insecurity (implemented February 2019 at our institution). These fields are found in the “Socioeconomic” section of the “Patient History” tab and have structured responses that range from “Never true” to “Often true” and include a “declined to answer” option. To measure the use of these fields for SDOH documentation (which was the focus of our study, rather than measuring the level of positively identified social need based on responses in these fields), we captured whether or not any field was populated with any response.

We selected 4 other data types relevant to social need data in the EHR to use as comparators. (Note: We did not attempt to capture all ways in which SDOH data may be captured in EHRs as this was not the objective of our study.) First, we selected the use of Z-codes on the EHR's problem list because it similarly offered a structured approach to capture specific social needs and was populated by physicians; however, unlike the new structured social history fields, they had been available since 2016, which allowed us to compare the use of a newer approach to an older approach. Second, we selected a housing status question included in a nurse flowsheet as a comparator that offered variation in who completes it (nurses), where in the EHR it is located (a flowsheet), and the level of workflow integration at our institution (high). Lastly, we selected social history free-text fields and social work notes that similarly targeted the capture of social needs data but are wholly narrative and only accommodate unstructured data. More detail on each data type is as follows.

Problem list

The EHR contains lists of clinical data such as diagnoses, procedures, or medical history and can also contain SDOH data. SDOH data on these lists identify specific social needs, which are facilitated when entries include dedicated social codes. For example, problem lists and diagnosis lists can include International Classification of Diseases, 10th Revision (ICD-10) Z-codes that were released in 2016 and are intended to capture factors that influence health status but are not directly medical. Z-codes are populated by the clinician if s/he has identified a social need for a patient. Our documentation measures capture the presence of a Z-code for 4 specific SDOH domains: unstable housing, food insecurity, transportation problems, or financial strain on the patient’s problem list (specific codes in Supplementary Appendix Table S1).

Inpatient nursing admission question

UCSF Health built a custom structured housing status question into the hospital admission flowsheet in June 2012, which is a set of questions administered by nurses to patients during hospital admission. As above, documentation measures for this data field capture whether or not this question was populated, regardless of actual response, in order to reflect use.

Patient social history text

Within the patient history section of the EHR, there is a free-text field labeled “social documentation” that was implemented in June 2016. Entries in this field can contain any type of text related to social needs or other social issues (eg, a description of the patient’s hobbies). Our documentation measures capture whether this free-text field was populated or updated, which again is a measure of use but does not specify the presence of a social need (which could only be established via text review).

Social work notes

While any type of clinical note could include SDOH information, there are specific types of clinical notes that are enriched for text relevant to SDOH. We focused on social work notes (ie, clinical notes written by social workers) because they typically capture social needs, such as housing instability and food insecurity. They have been available as a category of note type since our EHR was implemented in 2012. Our documentation measures capture the presence of one or more social work notes. As with the prior example, these measures indicate use but require text review to determine the presence of a social need.

Measure implementation context

Since the degree to which data fields are populated typically depends on their integration into standard clinical workflows, we also describe, based on study authors’ clinical practice experience, the extent to which each of our 5 data types captured documentation that was part of standard workflow at our institution during the study period. These workflow expectations offer critical contextual data for interpreting the measures. At UCSF Health, completing documentation and review of SDOH data was largely at the discretion of the clinician. Two exceptions included: (1) all pediatric inpatients were supposed to have a social work assessment and associated social work note completed within 48 h and (2) all inpatients were asked the inpatient nursing question within the hospital admission flowsheet (which were custom built for this purpose). Table 1 presents details about the 5 data types and associated workflows.

Table 1.

Overview of EHR data categories with examples of UCSF SDOH data documentation

EHR data type	Data type description	Part of expected documentation at UCSF	Date of UCSF implementation	Data type domains captured
				Food insecurity	Transport access	Financial strain	Housing stability	Any (free text)
Problem List	Problems on the Problem List section for unstable housing, food insecurity, transportation problems, or financial strain derived from ICD codes (see Supplementary Appendix for specific codes)	No	Oct 2016	x		x	x
Patient Social History Questions	Structured questions in the “Socioeconomic” section of the Patient History page that ask about food insecurity, transportation problems, and financial strain, and provide discrete answer options	No	Feb 2019	x	x	x
Inpatient Nursing Question	Housing status entered on nursing flowsheet during admission intake	Yes	June 2012a				x
Patient Social History Text	Free text in the “Social Documentation” section of the “Patient History” page	Specialty-Specific	Jan 2016					x
Social Work Note	Free-text clinical note written and signed by a social worker	Specialty-Specific	June 2012a					x

^aDate of EHR implementation.

Measures: units of analysis

Our 5 data types could be documented and reviewed for any set of patients and across any time period. Since we sought to develop measures that could be applied at the encounter level as well as the patient-year level, we chose a population that had at least one hospitalization in our 1-year study period. Using hospitalizations as the basis for our encounter-level measures allowed us to most robustly attribute documentation and review activities to a given encounter by bounding these activities by admission and discharge dates. (A similar, time-bound approach could have been used to capture encounter-level user activity in the ambulatory setting, but with more potential for misattribution since documentation and review activities do not occur within a standard time window around the encounter.) We then extended our measures to the patient-year level by bounding based on calendar date, resulting in measures that are encounter agnostic because they capture documentation and review activities that occurred over the course of the entire 1-year study period (including but not limited to hospitalization[s]). They therefore reflect SDOH documentation and review activities that occurred when patients were seen in any setting (eg, inpatient, ambulatory, emergency) over the course of the year and provide a perspective on the healthcare system’s overall use of the given type of SDOH data.

For sample patients, we extracted from Epic’s reporting database (Clarity) SDOH documentation metadata (from data element-specific tables) and SDOH review metadata (from audit log tables) during our sample period as well as patient demographics (eg, age, gender, insurance class, race/ethnicity) and the number and type of encounters during our study period.

Capturing encounter-level documentation for each data type

Within the context of a hospital encounter, we measured the level of documentation across our 5 data types in 2 ways. The first measure captures the percent of encounters in which SDOH data were documented during encounter (DDE), counting only data that were created or modified between encounter start and end date. Our second measure, documented by the time of discharge (DBD), expands on DDE and counts documentation that could have been entered at any point from 2012 (when UCSF Health’s EHR was implemented) through the encounter end date (ie, capturing whether the patient has any historical documentation by encounter end date). For example, for social work notes, our DDE measure is the percent of encounters where at least one social work note was created between the encounter admission time and discharge time; our DBD measure is the percent of encounters where at least one social work note was created before the encounter discharge time, including during a prior hospitalization or other encounter.

Full encounter-level measure specifications are included in Table 2. For patient-year analysis, DDE and DBD analogs are documented during study period (DSP) and documented by study period end (DSPE). Please refer to Supplementary AppendixTable S2 for specifications of patient-year level measures. If an encounter had multiple documentation events of the same type, it was only counted once when calculating the percent of encounters that had at least one documentation event. Similarly, if a patient had multiple documentation events in the patient-year, it was only counted once when calculating the percent of patient-years that had at least one documentation event. Though it represents a common unit of analysis for operational and quality use cases, encounter-level metrics may be skewed by a small set of patients with social needs who have multiple encounters. Patient-year metrics are therefore a useful complement because they are not subject to this oversampling limitation.

Table 2.

Measure specifications for inpatient encounter-level data types

	Hospital encounter
	Measure 1: Documented by discharge (Any time prior to discharge time) Measure 2: Documented during encounter (Between encounter admission and discharge time)	Measure 3: Reviewed during encounter Between encounter admission and discharge time
	Percent of encounters with…
Problem List	At least one social risk-related Z-code on the problem list	At least one Marked as Reviewed event of the problem list when a Z-code is populated
Patient Social History Questions	At least one structured SDOH question populated in the patient history	At least one Marked as Reviewed event in the social history section of the patient history when an SDOH question is populated
Inpatient Nursing Question	At least one housing question populated in the inpatient nursing admission flowsheet	Data are not available
Patient Social History Text	Social documentation free text populated in the patient history	At least one Marked as Reviewed event in the social documentation free-text section of the patient history when the section is populated
Social Work Notes	At least one social work note created	At least one social work note viewed

Capturing encounter-level review for each data type

To capture the frequency with which documented data for each data type is reviewed by care team members, we generated a reviewed during encounter (RDE) measure as the percent of encounters in which the data were reviewed at least once by any user during the hospital encounter. For patient social history questions, patient social history text, and the problem list, the EHR audit log captures when users click a button and indicate that they have reviewed the data. The audit log does not, however, capture times when these data types are displayed to users (ie, viewed) but the user failed to click the “Mark as Reviewed” button, which likely undercounts review events. In addition, for patient social history questions and the problem list, when a user clicks “Mark as Reviewed,” there is no way to know if they specifically reviewed the SDOH-relevant data (eg, whether they reviewed problems with the specific Z codes of interest). As such, review events were only counted toward the RDE numerator if at least one element of social data was available for review on the screen when users clicked “Mark as Reviewed.” For social work notes, users click to open the note and read it, generating an audit log entry that reflects which specific note was read. We therefore included all events representing the opening of one or more social work notes. As with documentation measures, review events were not double-counted within encounters or within patient-years. Review of inpatient nursing question data is not captured in the audit log and so it is not currently possible to measure review for this field. Full measure specifications are included in Table 2. The patient-year analog to RDE is reviewed during study period (RSP). Please refer to Supplementary AppendixTable S2 for specifications of patient-year level measures.

We also calculated an encounter review rate for each data type. This measure reflects the percent of encounters with data DBD that had data reviewed during the encounter (RDE). All analyses were performed using R version 3.5 and Stata version 15.1 (StataCorp). This study was approved by UCSF’s IRB (# 20-30268).

RESULTS

Our study sample included all patients who were admitted to a UCSF hospital at least once in the study period (n = 27 127 unique patients and 36 726 hospital encounters, Table 3). On average, patients had 1.36 hospitalizations (range 1–22).

Table 3.

Demographic and encounter characteristics of sample patients

	Sample
	(n = 27 127)
Encounters during study period
Inpatient encounters (mean # [SD])	1.36 (1.01)
Emergency encounters (mean # [SD])	2.33 (6.37)
Outpatient encounters (mean # [SD])	11.6 (22.8)
UCSF primary care provider (%)	0.218 (0.413)
Age on admission (mean in years [SD])	47.3 (24.7)
Gender (number [%])
Female	14 953 (55.1)
Male	12 164 (44.8)
Other/unknown	10 (0.0)
Race (number [%])
White or Caucasian	14 003 (51.6)
Other	5659 (20.9)
Asian	4270 (15.7)
Black or African American	2082 (7.7)
Unknown/declined	824 (3.0)
American Indian or Alaska native	283 (1.0)
Ethnicity (Number [%])
Hispanic or Latino	4745 (17.5)
Not Hispanic or Latino	21 539 (79.4)
Unknown/declined	836 (3.1)
Insurance class (number [%])
Private	9868 (36.4)
Medicare	8844 (32.6)
Medicaid/medical	6514 (24.0)
Self-pay	1003 (3.7)
Other	752 (2.8)

Note: For patients with multiple encounters, demographic information is from the most recent admission.

Encounter-level documentation and review measures

Applying the documentation measures at our institution revealed substantial variation: social work notes, patient social history text, and inpatient nursing question featured much higher levels of capture compared to patient social history questions and problem list (Figure 1). Specifically, the inpatient nursing question had the highest level of documentation during encounter (86.0% of encounters) and documentation by discharge (88.5% of encounters). Patient social history text had a DDE of 12.1% of encounters and a higher DBD of 49.0% of encounters (indicating that these fields were often completed during an encounter prior to the focal hospitalization). Similarly, social work notes had a DDE of 27.5% of encounters and DBD of 50.9% of encounters. In contrast, the problem list had a DDE of 0.09% of encounters and DBD of 0.59% of encounters. Finally, patient social history questions were DDE for 0.03% of encounters and documented by discharge for 0.26% of encounters.

Figure 1.

Hospital encounter SDOH documentation measures: UCSF Health. (A) Illustrated timelines of documentation measures. (B) Measures for our 5 selected data types.

We also found substantial variation in levels of review. Social work notes had the highest level of review; they were reviewed in 35.6% of encounters. There was an additional 15.2% of encounters in which social work notes were available but not reviewed, representing a review rate of 70.0% (35.6% reviewed/50.9% available). This was followed by patient social history text with an RDE of 14.4%, with 34.6% of encounters that had data available but not reviewed, representing a review rate of 29.4% (14.4% reviewed/49.0% available). While levels of documentation were very low for the problem list and patient social history questions, we still calculated the RDEs (0.1% and 0.03%, respectively) and review rates (21.7% and 11%, respectively) (Figure 2).

Figure 2.

Hospital encounter SDOH review measures: UCSF Health. (A) Illustrated timeline of review measure. (B) Measures for our 5 selected data types.

We report our equivalent measures for the patient-year unit of analysis in the Supplementary Appendix.

DISCUSSION

In the context of growing investments in capturing SDOH data in EHRs, we sought to develop measures of use that can be readily implemented across different SDOH data types, different units of analysis, and different settings to provide insights into the evolution of documentation and review of these data. The use of standard measures in studies examining SDOH documentation and review will foster more robust cross-institutional learning. We demonstrated the potential learnings that may be gleaned from these measures by applying them to 5 data types for a cohort of patients hospitalized over a 1-year period at a large, urban academic medical center. Results revealed substantial variation across our selected data types in how often data were documented and reviewed. In particular, the inpatient nursing question on housing status was most frequently documented (DBD for nearly 90% of encounters), followed by social work notes and patient social history text (DBD in approximately half of encounters). These levels of documentation stood in contrast to almost negligibly low levels for the problem list and patient social history questions. Finally, we found that once data were documented, they were often not reviewed by other clinical team members, though again the levels of review varied across data types. Although our data review measures only captured review that results in an audit trail, these findings nonetheless suggest that a large push focused solely on SDOH data documentation may still fail to impact care because SDOH information is not routinely reviewed.^19,20

Our results suggest that institution-specific workflows likely play a major role in the level of use of SDOH fields. The inpatient nursing question was by far the most highly used (from a documentation perspective) and also the only field that was part of standard workflow for all inpatient settings. This is particularly notable given that the other structured fields we examined were used at such low levels, and the use of this field even exceeded that of narrative data types. While such a relationship is not surprising, it nonetheless highlights the need to build and implement new workflows around the documentation and review of high-priority SDOH data. Notably, patient social history questions, designed explicitly to increase documentation of SDOH, were rarely used even 1 year after their introduction at our health system. We (and others^11,12) have found that another structured approach—adding Z-codes to the EHR's problem list—also did not result in high levels of SDOH data documentation even many years after these codes were released. This suggests that simply adding dedicated SDOH fields and tools to EHRs is insufficient to drive increased documentation of social needs, a conclusion consistent with other publications.^13,21 Thus, in addition to making fields available, education, workflow redesign, incentives, and institutional policy may be needed to increase uptake. As we experiment with such approaches, our measures can be leveraged to signal whether these efforts are having the intended effect, thereby providing a feedback loop to facilitate broader learning.

Our measures may also be useful to inform the investigation of the optimal cadence of SDOH documentation. When DDE nearly matches DBD (as is the case with the inpatient nursing question), this suggests that data types are being collected nearly every encounter. This may be appropriate for SDOH data that change frequently and thus should be updated every encounter (eg, housing status, insurance status). In contrast, when DDE is significantly lower than DBD (as is the case with patient social history text), this indicates that a data type is updated less often. This may be most appropriate for SDOH data that change every few years (eg, education history) or does not change once documented (eg, adverse childhood events for adults). Today, there are minimal standards or other guidance for how often (and in which setting[s]) each type of data should be captured/updated.

More broadly, our results underscore the need for more information about best practices—including EHR tools that facilitate data review—and better communication with clinical teams around expectations for SDOH data documentation and review. This may also require prioritizing some SDOH elements and including them in standard workflows, such as nursing and social work documentation. As an example, at our institution, there are multiple locations in which homelessness can be documented in the EHR and no current way to make all relevant data visible to subsequent users across these different data fields. Increased investment in surfacing data previously documented, particularly from a highly populated data field (ie, the inpatient nursing question), across multiple providers’ workflows would reduce the time needed by clinical staff to find information about homelessness; indeed, our institution is working to automatically import homelessness data from the inpatient nursing question into physician notes. Other EHR features, such as keyboard shortcuts and note templates that enable data imports, can facilitate streamlined SDOH documentation as well as enable more opportunities for SDOH data review.

Our measures should be considered in the context of several important limitations. First, our measures are not designed to quantify the totality of SDOH information in the EHR. Instead, they are designed to measure 2 dimensions of use that can be applied to different types of EHR fields and, when applied to fields relevant to SDOH data, help monitor uptake. Relatedly, SDOH data review measures (for patient social history questions, patient social history text, problem list) likely include some measurement error, though the error could go in either direction: levels could be underestimated because our data do not capture viewing if the data are not actively marked as reviewed or overestimated because we do not know if SDOH-specific information itself was reviewed when other data are contained in the same fields. Second, the availability of audit log data may be restricted in some settings (eg, not stored for long time periods because of amount of storage space required), extraction of audit log data can be computationally burdensome and complex, and specific user activities captured by the audit log may vary from institution to institution. Nonetheless, these data are routinely captured (due to Health Insurance Portability and Accountability Act requirements), typically available (since they are used for compliance), and fairly simple in structure. Third, our measures do not shed light on ideal or target values for the measures we develop. There are no standards for the levels of documentation or review of SDOH data, or consensus about the health professionals (eg, advanced practice clinicians, behavioral clinicians, or other staff) that should be engaging in these activities. Nonetheless, our measures and others like them facilitate important efforts toward establishing relevant standards. A final limitation specifically relevant to interpreting UCSF measure values is that, during our study period, a California law requiring reporting of homelessness status for all admitted patients was implemented. While this could have led to the high levels of inpatient nursing question documentation we observed, the low use of housing Z-codes suggests that it did not broadly drive high levels of SDOH documentation.

CONCLUSION

In response to health systems’ efforts to better integrate medical and social data, EHRs have expanded options for documenting patients’ SDOH information. To date, there has not been a standard approach to such integration, resulting in experimentation with different documentation options and different organizational strategies to promote uptake. We developed a generalizable set of measures that are useful to assess uptake of new structured fields and can also be applied to other EHR data types that are typically available to document SDOH data. When we used data from our institution to implement measures for the newly available structured SDOH fields and 4 comparator data types, we found significant variation in both documentation and review, with the lowest levels of documentation in the structured fields that were not part of expected documentation. These measures can guide optimization efforts, such as health professional education and workflow redesign, that might increase uptake, thereby ensuring that health systems have SDOH data available to inform clinical decision-making.

FUNDING

This work was supported by a gift from The Lisa & John Pritzker Family Fund. MSP also receives support from the Agency for Healthcare Research and Quality under Award Number K12HS026383, and the National Center for Advancing Translational Sciences under Award Number KL2TR001870.

AUTHOR CONTRIBUTIONS

All authors contributed to the design and writing of the manuscript. MW and MP are responsible for the accuracy of the analyses. MW is responsible for the final contents of the manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.