Natural language processing-driven state machines to extract social factors from unstructured clinical documentation

Abstract

Objective

This study sought to create natural language processing algorithms to extract the presence of social factors from clinical text in 3 areas: (1) housing, (2) financial, and (3) unemployment. For generalizability, finalized models were validated on data from a separate health system for generalizability.

Materials and Methods

Notes from 2 healthcare systems, representing a variety of note types, were utilized. To train models, the study utilized n-grams to identify keywords and implemented natural language processing (NLP) state machines across all note types. Manual review was conducted to determine performance. Sampling was based on a set percentage of notes, based on the prevalence of social need. Models were optimized over multiple training and evaluation cycles. Performance metrics were calculated using positive predictive value (PPV), negative predictive value, sensitivity, and specificity.

Results

PPV for housing rose from 0.71 to 0.95 over 3 training runs. PPV for financial rose from 0.83 to 0.89 over 2 training iterations, while PPV for unemployment rose from 0.78 to 0.88 over 3 iterations. The test data resulted in PPVs of 0.94, 0.97, and 0.95 for housing, financial, and unemployment, respectively. Final specificity scores were 0.95, 0.97, and 0.95 for housing, financial, and unemployment, respectively.

Discussion

We developed 3 rule-based NLP algorithms, trained across health systems. While this is a less sophisticated approach, the algorithms demonstrated a high degree of generalizability, maintaining >0.85 across all predictive performance metrics.

Conclusion

The rule-based NLP algorithms demonstrated consistent performance in identifying 3 social factors within clinical text. These methods may be a part of a strategy to measure social factors within an institution.

INTRODUCTION

Social factors, that is, an individual’s housing, food, employment, and income situations, affect health outcomes, utilization, and costs.^1–3 As a result, data on patients’ social factors have application to clinical decision-making, organizational-level planning and decision-making, and research studies.^1,4 There is emerging interest in capturing social factor data via social screening questionnaires and structured diagnostic codes (ie, ICD-10 and LOINC).⁵ However, a rich variety of social factor data may already be routinely documented in clinical free-text notes collected by clinical and nonclinical staff.^6,7 The goal of this project is to develop automated approaches to identify key social factors from multiple types of free-text notes collected from different patient populations. We created and validated natural language processing-driven state machines to detect 3 types of social factors: housing instability, financial insecurity, and unemployment.

Background and significance

Clinical care has a long tradition of recording social factors and risks within patients’ health records as free-text data.^8,9 Social history is a standard portion of health records.¹⁰ Social risk factor information appears in clinical notes from different medical specialties¹¹ and it could be in the form of products of services delivered by physicians, nurses, or social workers.⁷ However, the unstructured nature of text creates well-known challenges in reuse for clinical decision-making, aggregate reporting, or as input for other informatics tools, such as referral systems.

A potential solution to extracting person-level social factors from the clinical notes is natural language processing (NLP).¹² NLP has been successfully applied to clinical conditions including cancer,¹³ cardiovascular issues,¹⁴ and mental health.¹⁵ Nevertheless, the use of NLP to identify social factors may pose greater challenges. For example, documentation of social factors in clinical text is highly variable between clinicians.¹⁶ Also, different healthcare professionals may describe the same social concept using inconsistent language, or the concept may be inconsistently described in different types of clinical notes.¹⁷ Additionally, social factors are complex, intertwined, and often recounted by patients as stories or narratives, which may be difficult to record within EHRs.¹⁸ All these factors complicate extraction.

Social factors are not a completely new domain for NLP research. However, previous NLP work has tended more towards the identification of socio-behavioral risks than social factors, per se. For example, a recent systematic review concluded that the identification of tobacco use, alcohol use, substance abuse, physical activity, or sexual activity constituted the majority of publications on NLP.¹² Specific to social factors, numerous publications have used NLP to identify homelessness,^7,19–23 which is the most acute manifestation of housing instability.²⁴ However, efforts to identify factors such as financial insecurity and employment are much less frequent.¹² Moreover, previous work has focused primarily on NLP within a specific context. For example, the work will focus on one condition/cohort, such as substance abuse.^25,26 Other work relies heavily on existing, freely available clinical text (eg, the Medical Information Mart for Intensive Care dataset [MIMIC]),^27–29 certain note types or specific sections of notes (eg, social work/social history),^27,29 or utilizes surveys collected for nonclinical care purposes.³⁰ Studies looking at the generalizability across health systems are rare, with limited evidence to date. However, recent work has examined the ability to identify housing insecurity across multiple healthcare systems.²³

This study is motivated by the current state of applied NLP research as outlined above and seeks to contribute to the literature in 3 ways: (1) identifies a methodology applied to multiple social factors, with a focus on understudied factors related to economic stability, (2) is applied to a broad population representing multiple clinical settings and note types, and (3) focuses on the generalizability of the NLP algorithms to outside health systems.

MATERIALS AND METHODS

We developed a series of NLP-driven state machines³¹ to identify social risk factors present in clinical notes obtained from 2 different health systems based in Indiana, USA. Development included a multistep, systematic, iterative process with keyword and rule identification, state machine development, and expert validation.

Social factors

We sought to identify the following highly prevalent social risk factors: (1) housing instability, (2) financial insecurity, and (3) unemployment. All 3 risk factors have been associated with economic stability, which is a central concern of Healthy People 2030.³² Housing instability is estimated to affect 20% of households in the United States.³³ The connection between housing instability and clinical care is extremely critical as issues related to housing have been shown to share risk factors with physical trauma³⁴ as well as health impacts from unstable or inadequate housing.³⁵ Evidence shows that income shapes health behaviors³⁶ as well as the ability to seek appropriate medical care.³⁷ Economic stability, or its inverse of instability, may change behaviors in a myriad of ways and clinician awareness of these social factors may help shape the care provided. For this study, the 3 constructs were defined as follows:

• Housing instability: Any housing disruption or housing-related problem such as frequent moves, difficulty paying rent, eviction, or homelessness.³⁸

• Financial insecurity: A situation in which a person cannot fully meet current and ongoing financial obligations without fear of not having enough money to cover necessary expenses.^39,40

• Unemployment: Individuals who are not working, but eligible to participate in the workforce. This excludes individuals who are students, homemakers, or disabled.⁴¹

Sample and data

The primary data source was the Indiana Network for Patient Care (INPC).⁴² INPC is a regional health information exchange with multiple participating health systems. For the purposes of this study, clinical notes were isolated from 2 Indianapolis, IN area health systems that represented high-volume, diverse clinical populations: one is a public safety-net provider, and the second is a nonprofit system serving largely privately insured individuals. We purposefully selected these different sources to support model generalizability. Clinical notes included all clinical documentation shared with INPC, which includes a variety of note types (see Supplementary Appendix S1 for most common note types). The training corpus of notes was obtained from a multihospital health system representing urban and rural settings. Notes associated with any type of clinical encounter were included if they were created between January 1, 2019, and December 31, 2019 (n = 1 710 124 notes, 581 205 unique patients). The test corpus of clinical notes was extracted from any clinical encounter for a second, separate health system. These notes were obtained from a safety-net hospital with multiple health clinics that were documented between September 1, 2020, and March 31, 2021 (n = 724 308 notes, 74 239 unique patients). Notes did not undergo any preparation processes and were utilized in their raw form. Clinical notes were linked via patient identifiers to clinical and demographic information from INPC. These measures included age, gender, race, ethnicity, rural/urban status, Modified Townsend Index (social deprivation based on residential zip code),⁴³ and Charlson Comorbidity score.⁴⁴

Keyword identification

Utilizing 1 month of notes from the training set, we produced a list of continuous sequence words (n-grams)³¹ for review. This list was filtered utilizing Term Frequency-Inverse Document Frequency (TF-IDF) measurements, to identify and remove stop words not contributing value (eg, “this”, “what”). Team members reviewed 1-, 2-, and 3-grams to determine any connection to housing instability, financial insecurity, or unemployment. Each n-gram was reviewed independently by 2 research assistants with differences adjudicated by the authors. We also identified potential keywords from published social risk factor screening questionnaires.^45,46 The result of this step was an initial set of keywords for inclusion in each state machine.

State machine development

We applied a deterministic finite state machine methodology,⁴⁷ which has been previously proven to work with clinical data,⁴⁸ using a multistep development and validation processes (see Fig. 1). These processes identified clinical notes with mentions of housing instability, financial insecurity, or unemployment. We constructed and validated the machines independently from one another, utilizing the Regenstrief natural language processing environment nDepth^49–51 to implement the state machines. nDepth is a platform that enables logical keyword-based searches, extraction of structured information such as n-grams and processing pipelines for text segmentation and tokenization (state machines), and manual validation of pipeline performance. nDepth indexes and searches the corpus of clinical notes using the keyword dictionary and associated rules (eg, negation).^49–51Figure 1 provides an overview of our approach.

Figure 1.

State machine process diagram.

First, we constructed individual initial state machines for each of the 3 social factors utilizing the terms generated in the keyword identification phase. For each machine, 2 reviewers (KA and DH) manually reviewed 200 randomly selected notes from the training corpus to assess machine performance using positive and negative predictive values (PPV and NPV). This was utilized to determine the initial performance of each machine and to identify keyword adjustments and any issues related to language (eg, template language containing keyword or alternate meaning to keyword).

Next, we retrained the state machine on a new, larger set of training corpora of notes, derived from the original set of notes identified previously. This iterative step included adjustments to each state machine based on how each term performed and reviewer comments. The state machines were again reassessed via manual review. We continued this process, gradually updating each state machine based on expert validation results. Updates to the state machines included the addition of new keywords as well as the creation of new rules such as (1) the definition of formal search spaces (eg, search for the keyword “sleep” within 5 words of the word ‘car’, (2) grammatical rules to identify negation (eg, the patient is able “to afford” vs unable “to afford”), and (3) to identify context (eg, differentiate between the use of the phrase “does not work” as it applies to a clinical concept such as ‘medication does not work’ vs how it applies to an individual who is not employed).

Given the low prevalence of reported social factors in the notes, we focused on a subset of note types where the social factors were most commonly identified (see Supplementary Appendix S1). For example, the more common note types included progress and visit notes and we ultimately excluded documents related to pathological reports, which do not typically have social history. For subsequent training iterations, we identified the top 10 most common note types, and 4 reviewers manually examined an equal number of positive and negative results, matched on note type. This is similar to methods deployed in other studies requiring manual review.⁵² We completed the iteration process if the state machine reported PPV and sensitivity scores ≥0.85.

To ensure consistency, the reviewers annotated an overlapping subset of 50 notes from each of the 3 state machines. Reviewers exhibited a high level of agreement (see Supplementary Appendix S2).

Finally, we applied each of the finalized state machines to the test corpus of notes from the second health system. Manual review was completed per the same methodology as the second step. The state machine was considered finalized if PPV and sensitivity were ≥0.90.

Analysis

Frequencies, percent, means, and standard deviation described the study samples. PPV, NPV, sensitivity, and specificity described the performance of each state machine. The Indiana University Institutional Review Board approved this study.

RESULTS

Cohort characteristics

The corpora of notes validated in this study included a total of 9907 unique patients across the 2 health systems (Table 1). Consistent in an urban US healthcare setting, the majority of patients were from urban zip codes and were female. Patients in the test cohort tended to be more diverse and from more socially deprived areas, which is reflective of the safety-net role of the health system.

Table 1.

Overall cohort characteristics (persons)

	Total cohort characteristics (N = 9907)	Training corpus (n = 3621)	Testing corpus (n = 6286)
Age, mean (SD)	48.72 (17.21)	51.47 (18.58)	47.14 (16.42)
Female, n (%)	5316 (53.7)	1871 (51.7)	3445 (54.8)
Race, n (%)
White	5519 (55.7)	2922 (80.7)	2597 (41.3)
Black/African American	3320 (33.5)	487 (13.4)	2833 (45.1)
AAPI	177 (1.8)	25 (0.7)	152 (2.4)
Multiracial	132 (1.3)		132 (2.1)
Other	759 (7.7)	187 (5.2)	572 (9.1)
Hispanic	1131 (11.4)	81 (2.2)	1050 (16.7)
Geographic characteristics
Urban, n (%)	8933 (90.2)	2799 (77.3)	6134 (97.6)
Modified townsend, mean (SD)	1.78 (4.33)	0.11 (4.51)	2.68 (3.90)
Clinical characteristics, mean (SD)
Encounters PPa	19.2 (20.69)	13.8 (1.30)	22.32 (24.33)
Notes PPb	30.67 (35.45)	15.36 (13.23)	39.48 (42.22)
Charlson comorbidity score	1.84 (2.51)	1.63 (2.44)	1.93 (2.53)

^aEncounters PP: Statistics related to the number of clinical encounters per person in the data set.

^bNotes PP: Statistics related to the number of clinical notes per person in the data set.

State machine performance

The state machines identified a prevalence of social factors which ranged from 0.06% to 1.54%. This prevalence rate suggests a low baseline of documentation but should be interpreted cautiously as the state machine was run across all notes, including imaging, which is less likely to contain mentions of social factors. Table 2 shows all results for the initial training run, final training run, and the performance of each machine on the final holdout test set of notes. Initial state machine results, based solely on the identified keywords, were promising with moderate PPV and excellent NPV. Repetitive iterations with the training data resulted in increased PPVs, meeting our criteria for movement to test data, and excellent sensitivity results.

Table 2.

Validation results by state machine (note level)

	Housing	Financial	Unemployment
Training run 1a	n = 200	n = 200	n = 200
PPV	0.71	0.83	0.78
NPV	1.00	1.00	1.00
Training run 2	n = 200	n = 1025	n = 1980
PPV	0.89	0.89	0.83
NPV	1.00	0.99	0.99
Sensitivity	1.00a	0.99	0.99
Specificity	0.90a	0.9	0.85
Training run 3	n = 997	n/a	n = 200
PPV	0.95	—	0.88
NPV	1.00	—	0.98
Sensitivity	1.00	—	0.94b
Specificity	0.95		0.96b
Test run (final)	n = 997	n = 2136	n = 4110
PPV	0.94	0.97	0.95
NPV	1.00	0.98	0.99
Sensitivity	1.00	0.98	0.99
Specificity	0.95	0.97	0.95

^aRepresents the first training iteration with all keywords and no modifications.

^bShould be interpreted cautiously given the artificially constrained sample size.

Overall, the housing instability state machine required 3 training iterations, financial insecurity 2 training iterations, and unemployment 3 training iterations to reach the threshold sensitivity threshold score (>0.85) we defined. Specificity was lower but remained over the 0.85 threshold. Following slight adjustments, the final state machine performance, conducted on the test corpus of notes, was strong. PPV, sensitivity, and specificity were above 0.90 for all state machines.

DISCUSSION

We developed 3 NLP-driven state machines using free-text notes captured from a health system that served a commercially insured population and demonstrated their generalizability to a health system serving an underserved patient population. By developing and calibrating models across health systems that serve different patient populations, we demonstrated high degrees of model generalizability, which is critical in working with multi-institutional data, such as health information exchange systems.

Methodologically, this study highlights 2 challenges often identified in applied healthcare analytics. The first is generalizability. The ability of models, NLP or otherwise, developed using data from one population to deliver satisfactory predictive performance across other previously unseen, statistically different patient populations, is an ongoing challenge.⁵³ For example, some projects targeting social factors identification have focused on vulnerable populations (eg, substance abuse, HIV patients).^25,26 The advantage of using these targeted cohorts is that the higher likelihood social factors are frequently documented due to their known associations with health outcomes. While this may make statistical model or algorithm development more feasible, generalizability to a less vulnerable population is uncertain. For this study, we were able to leverage 2 healthcare institutions, that, while in the same geographic area, generally serve 2 different patient populations, that is, the commercially insured and the underserved. This is a more rigorous approach towards establishing generalizability.

The second methodological issue was our choice of a state machine-based approach. A majority of analytical efforts involving free-text datasets have shifted towards complex, resource-intensive approaches such as neural networks and deep learning²⁹ to identify and classify various social factors.^29,54 While these models may yield superior performance under test settings, they are complex approaches that require a high degree of technical expertise, greater computing resources, and present scalability issues. In any machine learning application, researchers and practitioners must make choices between the tradeoffs of performance, implementability, and maintainability.^7,26,29,54 While these simple, finite-state machine methods are not as sophisticated as neural network-based approaches, they have several advantages. They are less complex, and thereby easier to develop, interpret, implement, and maintain.⁵⁵ In many cases, rules-based systems are more transparent, easier to communicate to nonexperts, and therefore more easily implemented in other health systems. Past studies have shown that rule-based methods are more sensitive, which may be important for phenomena that are more rarely documented, such as social factors.²⁵ Due to their simplicity, rule-based methods may be more generalizable, and thus, report more consistent predictive performance across health systems. Our choice of a state-machine approach may address some of the limitations facing advanced analytical methods including the potential for bias in artificial intelligence which particularly impact more complex, black box approaches such as deep learning methods. Learning algorithms run the risk of incorporating underlying societal biases contained within medical datasets.⁵⁶ This potential for bias is increased with the perpetuation of 2 particular practices, namely the utilization of the off-the-shelf NLP programs that may not have been trained to utilize clinical data and the reliance on shared/public annotated note sets (eg, MIMIC).^27–29 The methods discussed in this study limit bias by utilizing human intervention in the development of state machines and by leveraging a unique set of clinical notes.

The demand from payers, policy-makers, and advocates for information on patients’ social factors and needs is substantial⁵⁷ and multiple approaches are requested to obtain this information. In recent years, coding standards for recording social risks as structured data within EHRs using ICD-10 or LOINC codes have advanced substantially. Nevertheless, these structured data are very underutilized in practice.⁵⁸ Additionally, health systems and researchers have integrated social screening surveys into EHR environments. However, evidence suggests that social screening is not an exceptionally common practice.⁵⁹ Our findings indicate that the use of NLP to analyze existing, routinely collected free-text reports may be a feasible approach for healthcare organizations and researchers to identify patients with documented social factors. As clinical notes represent a different data generation process than coding or screening surveys, NLP could be applied as part of an overall social health measurement strategy. It is important to not discard clinical text in favor of screening or other structured methods for data collection. However, social factors extracted via NLP could be utilized to impute missing survey results, augment survey data, or—given the ability to apply retrospectively—provide a longitudinal description of social factors. As products of a clinical encounter, these patient interactions and the information within clinical notes are important. However, it is also critical to remember that the text is, by nature, selective, filtered, and containing omissions (either left unrecorded by the provider or never volunteered by the patient).^60–62 A comprehensive health measurement strategy will include formalized screening as well as information garnered from clinical documentation.

Once identified, social factor data have wide application. These data can improve risk prediction models,^63–66 match patients to appropriate social services,^67,68 and illuminate underlying disparities in population health.⁶⁹ Free-text data constitute an important source for feature creation¹¹ and are one that generally contribute to better prediction models.⁷⁰ Additionally, unlike structured codes or patient surveys that cannot be collected retroactively, NLP algorithms can be applied to historical data, for times prior to the implementation of formalized screening processes. While still acknowledging the caveats and the limitations of clinical notes, NLP could provide both a more reasonable measure of longitudinal social risk as well as data for retrospective cohorts.

Limitations

While these NLP algorithms were developed and validated across 2 distinct health systems, performance may not be similar in other statistically diverse patient populations. Further analyses could push generalizability in the area of settings by validating performance in a different geographic location. Additionally, while the state machines achieved acceptable performance for the 3 target social factors, such an approach may not be feasible for other social factors such as transportation barriers, legal issues, or food insecurity. More sophisticated methods may be necessary for these social factors or to achieve even better performance than our existing models. In addition, as social factor screening becomes more common in clinical practice, it is possible that documentation practices or language will change, which will necessitate the revision and updating of the state machines. Finally, our efforts were susceptible to limitations caused by misspellings, errors present in the free-text data, and challenges caused by the use of abbreviations specific to the medical domain.

CONCLUSION

NLP algorithms demonstrated strong performance in identifying cases of financial insecurity, housing instability, and unemployment among adult patients. NLP could be a part of an overall social health measurement strategy.

FUNDING

This work was supported by the funding from the Indiana University Addictions Grand Challenge (PI: Dr. Peter Embi), a University initiative to respond to the addictions crisis.

AUTHOR CONTRIBUTIONS

JRV and KSA conceived of this study and were responsible for primary writing activities. KSA and DRH provided manual validation as well as keyword identification guidance. KSA also completed the analyses. JC provided technical guidance and programming requirements. SK and EAM provided subject matter expertise, overall guidance, and manuscript editing.

SUPPLEMENTARY MATERIAL

Supplementary material is available at JAMIA Open online.

CONFLICT OF INTEREST STATEMENT

None declared.

Data Availability

Due to the ethical considerations, data and detailed analyses generated during the course of this study are available upon request. Access is subject to review and approval by the applicable Privacy Officers of the participating institutions. Contact corresponding author for assistance with request.