Natural language processing of prehospital emergency medical services trauma records allows for automated characterization of treatment appropriateness

Abstract

BACKGROUND:

Incomplete prehospital trauma care is a significant contributor to preventable deaths. Current databases lack timelines easily constructible of clinical events. Temporal associations and procedural indications are critical to characterize treatment appropriateness. Natural language processing (NLP) methods present a novel approach to bridge this gap. We sought to evaluate the efficacy of a novel and automated NLP pipeline to determine treatment appropriateness from a sample of prehospital EMS motor vehicle crash records.

METHODS:

A total of 142 records were used to extract airway procedures, intraosseous/intravenous access, packed red blood cell transfusion, crystalloid bolus, chest compression system, tranexamic acid bolus, and needle decompression. Reports were processed using four clinical NLP systems and augmented via a word2phrase method leveraging a large integrated health system clinical note repository to identify terms semantically similar with treatment indications. Indications were matched with treatments and categorized as indicated, missed (indicated but not performed), or nonindicated. Automated results were then compared with manual review, and precision and recall were calculated for each treatment determination.

RESULTS:

Natural language processing identified 184 treatments. Automated timeline summarization was completed for all patients. Treatments were characterized as indicated in a subset of cases including the following: 69% (18 of 26 patients) for airway, 54.5% (6 of 11 patients) for intraosseous access, 11.1% (1 of 9 patients) for needle decompression, 55.6% (10 of 18 patients) for tranexamic acid, 60% (9 of 15 patients) for packed red blood cell, 12.9% (4 of 31 patients) for crystalloid bolus, and 60% (3 of 5 patients) for chest compression system. The most commonly nonindicated treatment was crystalloid bolus (22 of 142 patients). Overall, the automated NLP system performed with high precision and recall with over 70% of comparisons achieving precision and recall of greater than 80%.

CONCLUSION:

Natural language processing methodologies show promise for enabling automated extraction of procedural indication data and timeline summarization. Future directions should focus on optimizing and expanding these techniques to scale and facilitate broader trauma care performance monitoring.

LEVEL OF EVIDENCE:

Diagnostic tests or criteria, level III.

In 2016, the National Academies of Sciences Engineering and Medicine identified that, of the approximately 150,000 annual trauma deaths, 20% were preventable.¹ Incomplete prehospital care is one of the largest contributors of preventable trauma mortality in the United States.^2,3 While valuable for understanding care before hospitalization, manual prehospital chart review and timeline recreation by trained clinical data abstractors is a time intensive and costly process. Furthermore, many trauma centers use trauma medical directors (TMDs) and trauma program managers (TPMs) to manually review prehospital charts for performance review. Text mining and natural language processing (NLP) techniques are computerized techniques used to automatically extract structured data from unstructured or semistructured notes, such as emergency medical services (EMSs) reports.⁴ These techniques hold promise since NLP can automate data abstraction allowing flagging of notes with potential gaps in care for downstream manual review by TMDs and TPMs. This can be beneficial and timesaving, so TMDs and TPMs can focus on notes with potential quality gaps.

The purpose of this pilot study is to evaluate the efficacy of a novel and automated application leveraging an NLP pipeline to determine treatment appropriateness from prehospital EMS motor vehicle crash (MVC) records. To the best of our knowledge, this is the first attempt to use NLP in the prehospital trauma setting. Our hypothesis is that the automated system will accurately estimate treatment appropriateness compared with manual chart review.

PATIENTS AND METHODS

Data Collection

This study used deidentified electronic MVC prehospital EMS run reports from a large metropolitan hospital-based EMS agency between February 1, 2017, and October 4, 2018. Patients who were treated by EMS but not admitted to the hospital (an American College of Surgeons verified level 1 trauma center) and therefore not included in the hospital trauma registry were excluded. In addition, patients who received treatment in the field but refused transport, patients who were dead upon arrival of EMS resources, or patients with traumatic arrest who had termination of resuscitation in the field per the EMS agency’s protocols were excluded. Interfacility transports between hospitals and EMS runs where multiple patients were injured and received care in a single report were also excluded. A total of 142 reports met these criteria and were included in the analysis (Fig. 1). Emergency medical service run reports were managed by the EMS agency’s quality resource supervisor and were generated from the EMS electronic patient care reporting system (Imagetrend Elite, Lakeville, MN). The electronic patient care reporting system allows for the generation of deidentified reports that are free of protected health information. The majority of information within each report was within an unstructured field of the note titled “Narrative,” yielding a large variety of clinically relevant information. This study was approved by both the University of Minnesota Medical School and North Memorial Health Hospital Institutional Review Boards.

Figure 1.

Study diagram detailing selection of patients.

NLP Information Extraction

We have previously reported an NLP methodology to extract named entities from unstructured text from prehospital trauma EMS reports.⁵ In brief, a criterion standard corpus consisting of two parts, schema creation and manual text annotation, was created for 37 clinically relevant entities. Clinical entities without discrete field representation in EMS and trauma registries were selected for extraction. While the schema included a number of other elements around mechanistic triage criteria and extent of the trauma, we focused upon indicated treatments in this analysis. Entities used for the analysis in this project included procedure (or treatment) and procedural (or treatment) indications. Treatments of interest included airway insertion, intravenous (IV) access, intraosseous (IO) access, needle decompression, tranexamic acid (TXA), packed red blood cell (PRBC) infusion, crystalloid bolus, and mechanical chest compression system for patients requiring cardiopulmonary resuscitation (CPR). Four widely used clinical NLP systems, cTAKES,⁶ CLAMP,⁷ BioMediCUS,⁸ and MetaMap,⁹ were then evaluated to determine the best-at-task union ensemble of system annotations for automated annotation of treatment and treatment indication elements. A timeline was then generated for each record facilitating automated summarization.

This novel ensemble of NLP systems was then augmented using a phrase2vec model we have previously developed and validated to identify terms semantically similar with treatment and treatment indications of interest.¹⁰ We used a cosine distance threshold of 0.7 for similarity matching using the phrase2phrase method with our phrase2vec model. This allowed the selection of semantically similar terms for a given list of treatment indications (unresponsive, unconscious, agonal, hypotensive, tachycardic, diminished breath sounds, absent breath sounds, desaturation, CPR, and massive hemorrhage) and treatments (ET tube, IV, IO, PRBC, transfusion, TXA, LUCAS, I-gel, airway, intraosseous, intravenous access, tranexamic, and bolus). This approach is beneficial, since it allowed for the capture of semantically similar words and phrases in the event that variation in documentation practices exists between EMS providers. For example, the system identified that pulselessness, chest compressions, and asystole were semantically similar to a patient receiving CPR. It also allowed for the extraction of more complex associations. For example, the system associated a patient requiring a sternal rub as semantically similar to being unresponsive.

Definition of Indicated Treatments

The following indications were used for each treatment:

• Airway intervention—indicated if nonresponsive, agonal respirations, or Glasgow coma scale less than 9.

• For the purpose of this analysis, an attempt at IV access was indicated for all patients.

• Intraosseous access—indicated if failed IV attempt, loss of IV access, or for patients requiring CPR.

• Needle decompression—indicated in hypotensive patients with diminished or absent breath sounds.

• Tranexamic acid—indicated in all hypotensive (systolic blood pressure [SBP], <90 mm Hg) patients (per local protocol).

• Packed red blood cell infusion—indicated in all hypotensive patients (SBP, <90 mm Hg) receiving air medical transport.

• Crystalloid bolus—indicated for hypotensive patients when blood is not available (ground transport) or for continued hypotension after both air medical transport blood units have been transfused.

• Mechanical chest compression system—indicated for all patients receiving CPR.

Treatments, treatment indications, and extracted examples of treatment indications and mapped NLP concepts are shown in Table 1.

TABLE 1.

Treatment, Treatment Indications, and Illustrative Examples of Treatment Indication NLP Text and Concept Association

Treatment	Treatment Indication	Examples ofNLP Text Extraction From Note	NLP Concept Associated With Text Extraction
Airway intervention (includes BVM, NPA, OPA, supraglottic airway, and endotracheal intubation)	Nonresponsive Agonal respirations GCS <9	Assisted, as PT agonal resps Chief (primary)|unresponsive—trauma PT is unresponsive, slumped over with snoring	Agonal Agonal Unconscious
IV access	All patients	N/A
IO access	Failed IV attempt Loss of all IV access Patients requiring CPR	IV was pulled out, IO placed.	IO
Needle decompression	Hypotension and Diminished or absent bilateral breath sounds	Breath sounds-absent-left, breath sounds-absent Contusion, breath sounds-decreased right, breath	Breath sounds Diminished breath sounds
Tranexamic acid PRBC infusion Crystalloid bolus	Hypotension Hypotension Hypotension when blood is not available Continued hypotension after PRBC	Hypotensive 80s SBP PT continued to have hypotension Due to continued low BPs	Hypotensive Hypotensive Hypotensive Hypotensive
Mechanical chest compression	All patients receiving CPR	PT in cardiac arrest, CPR started by bystander within 2 ECG monitor|PEA	CPR  CPR

BVM, bag valve mask; NPA, nasopharyngeal airway; OPA, oropharyngeal airway; GCS, Glasgow coma scale; PT, patient; ECG, electrocardiogram; PEA, pulseless electrical activity.

For the purposes of this study, a treatment was considered indicated if the indication was present and the patient received the treatment, missed if the patient had the indication for it and did not receive the treatment, and nonindicated based on available documentation if the patient did not have an indication and received a treatment. This study did not seek to determine the actual real clinical appropriateness of specific interventions but instead was designed to detect potential indication-treatment patterns in an automated fashion based on the documentation the EMS providers included in their electronic patient care report. A Stata-MP (version 15; StataCorp, College Station, TX) application was developed to determine treatment appropriateness and generate an automated report for each record.

Manual Comparison

All 142 MVC records were then manually reviewed, and study-defined treatment appropriateness was determined for each record using the same criteria outlined previously. Manual reviews were conducted by a board-certified surgical critical care trauma attending surgeon at our institution. Automated results were then compared with manual results, and precision (positive predictive value) and recall (sensitivity) were calculated for each treatment using Stata-MP.

RESULTS

A total of 142 MVC records were included in this analysis (Fig. 1). Overall data documentation was of very high quality with significant redundancy surrounding clinical status as related to treatment indications. There were 1,256 documentations related to procedural indications; this number includes documentations of an indication for a treatment (e.g., agonal respirations) or documentation supporting that a treatment is not indicated (e.g., patient alert and oriented ×3 and protecting their airway). For example, there were 437 documentations related to mental status, which corresponds to three documentations related to their mental status and its progression through EMS transport per note. In addition, there was 646 documentations related to patient breath sounds and saturations and 141 documentations of aberrant vital signs. There were 497 documentations related to treatments.

Automated timeline summarization was performed for each patient (Fig. 2). From manual review, there was a total of 184 indicated treatments delivered, 44 missed treatments, and 29 nonindicated treatments provided for the 142 patients in this study (Table 2). Ninety-four percent (133 of 142 patients) of patients received a peripheral IV. Only 69% (18 of 26 patients) of patients received indicated airway management, 54.5% (6 of 11 patients) received indicated IO access, 11.1% (1 of 9 patients) received indicated needle decompression, 55.6% (10 of 18 patients) received indicated TXA, 60% (9 of 15 patients) received indicated PRBC, 12.9% (4 of 31 patients) received indicated crystalloid bolus, and 60% (3 of 5 patients) received indicated chest compression system. The most commonly nonindicated treatment was crystalloid bolus (22 patients).

Figure 2.

Automated timeline summarization via NLP of prehospital trauma EMSs reports. BP, blood pressure.

TABLE 2.

Summary of Manual Review of Prehospital Treatment Appropriateness for 142 MVC Patients

	Indicated Treatment, n (%)	Missed Treatment, n (%)	Nonindicated Treatment, n (%)
Airway	18 (69.2)	7 (26.9)	1 (3.8)
IV access	133 (93.7)	9 (6.3)	0 (0)
IO access	6 (54.5)	2 (18.2)	3 (27.3)
Needle decompression	1 (11.1)	7 (77.8)	1 (111)
Tranexamic acid	10 (55.6)	6 (33.3)	2 (1.1)
Blood infusion	9 (60)	6 (40)	0 (0)
Crystalloid bolus	4 (12.9)	5 (16.1)	22 (71)
Chest compression system	3 (60)	2 (40)	0
Total	184	44	29

Airway: Automated vs. Manual

The automated system identified 15 patients who received an indicated airway, 17 missed, and 1 nonindicated compared with 18 indicated, 7 missed, and 1 nonindicated identified during manual chart review (Table 3). Precision was 1.0 in determining absence of both airway and indication, indicated airway, and nonindicated airway. Precision was low (0.41) in determining missed airway. Manual review identified that poor precision was due to a transient nature of unresponsiveness in approximately two thirds of disagreements and a lack of the NLP system to extract a documented airway intervention in one third of disagreements. Recall was greater than 0.8 for all airway comparisons.

TABLE 3.

Comparison of Automated Versus Manual Characterization of Treatment Appropriateness

		Manual
		None	Indicated	Missed	Nonindicated	Total	Precision	Recall
	Airway
System	None	109	0	0	0	109	1	0.94
	Indicated	0	15	0	0	15	1	0.83
	Missed	7	3	7	0	17	0.41	1
	Nonindicated	0	0	0	1	1	1	1
	Total	116	18	7	1	142
System	IV access
	Indicated		133	2		135	0.99	1
	Missed		0	7		7	1	0.78
	Total		133	9		142
	IO access
System	None	130	0	1	0	131	0.99	0.99
	Indicated	0	5	0	0	5	1	0.83
	Missed	1	1	1	0	3	0.33	0.5
	Nonindicated	0	0	0	3	3	1	1
	Total	131	6	2	3	142
	Needle decompression
System	None	133	0	0	1	134	0.99	1
	Indicated	0	1	0	0	1	1	1
	Missed	0	0	7	0	7	1	1
	Total	133	1	7	1	142
	TXA
System	None	120	0	1	0	121	0.99	0.97
	Indicated	0	8	0	0	8	1	0.8
	Missed	3	0	5	0	8	0.63	0.83
	Nonindicated	1	2	0	2	5	0.4	1
	Total	124	10	6	2	142
	PRBCs
System	None	124	0	1		125	0.99	0.98
	Indicated	0	8	0		8	1	0.89
	Missed	3	0	5		8	0.63	0.83
	Nonindicated	0	1	0		1
	Total	127	9	6		142
	Cystalloid bolus
System	None	107	0	0	4	111	0.96	0.96
	Indicated	0	2	0	0	2	1	0.5
	Missed	3	0	5	0	8	0.63	1
	Nonindicated	1	2	0	18	21	0.86	0.82
	Total	111	4	5	22	142
	Chest compression
System	None	136	0	0		136	1	0.99
	Indicated	0	3	0		3	1	1
	Missed	1	0	2		3	0.67	1
	Total	137	3	2		142

IV: Automated Versus Manual

The automated system identified 135 patients who received an indicated IV and 7 patients who did not (Table 3). Compared with manual review, the system was highly precise and sensitive to identify patients with indicated IVs (precision and recall, >0.99) but was only moderately sensitive to identify patients who did not receive IVs (precision, 1.0; recall, 0.78).

IO: Automated Versus Manual

The automated system identified five patients who received an indicated IO, three who were missed, and three who had nonindicated IOs place (Table 3). Overall, the automated system performed with high precision and recall for all elements (precision and recall, >0.83) except missed IO (precision, 0.33; sensitivity, 0.5). However, there was a low sample size of patients with missed IO (automated, 3; manual, 2).

Needle Decompression: Automated Versus Manual

Both the automated system and manual review identified one patient who received an indicated needle decompression and seven patients who were missed (Table 3). The automated system performed with greater than 0.99 precision and recall for needle decompression.

TXA: Automated Versus Manual

The automated system identified eight patients who received indicated TXA, eight who were missed, and five who received nonindicated TXA (Table 3). Precision and recall were high for patients who received indicated TXA (>0.8). Precision was poor for patients who were missed or received nonindicated TXA (0.4–0.63). Upon manual review of manual-system disagreements, it was identified that the automated system either did not extract that the patient was hypotensive or misinterpreted patients with pulseless extremities as having a systolic blood pressure of less than 90 mm Hg, which resulted in misclassification of the patients as hypotensive requiring TXA.

PRBC: Automated Versus Manual

The automated system identified eight patients who received indicated PRBC, eight who were missed, and one who received nonindicated PRBC (Table 3). Similar to TXA, precision and recall were high for patients who received indicated PRBC (>0.89). Precision was poor for patients who were missed, 0.63, because of similar issues regarding nonextraction of hypotension or pulseless extremities being identified as hypotensive.

Crystalloid Bolus: Automated Versus Manual

The automated system identified 2 patients who received an indicated crystalloid bolus, 8 who were missed, and 21 who received a nonindicated crystalloid bolus (Table 3). The system had high precision (0.86) and recall (0.82) to identify the delivery of nonindicated crystalloid bolus.

Chest Compression System: Automated Versus Manual

The automated system identified three patients who received indicated chest compression system during CPR and three who were missed (Table 3). The system had high precision and recall (>0.99) for patients not requiring CPR or had indications for the chest compression system. The system had moderate precision (0.67) and good recall (1.0) to identify missed patients.

DISCUSSION

This pilot study is one of the first studies using NLP artificial intelligence to characterize prehospital trauma treatment appropriateness to facilitate EMS performance review. We report the clinical utility of a novel clinical NLP ensemble pipeline for automated timeline summarization for prehospital EMS reports. On manual review, needle decompression was identified as the most frequently missed treatment, and crystalloid bolus was identified as the most frequently nonindicated treatment delivered. Automated characterization of treatment appropriateness performed well compared with manual characterization with an overall high precision and recall. While promising, further work is needed to understand if our findings are valid in other populations and to further develop these automated techniques to other centers to scale our knowledge of prehospital trauma care.

Studies suggest that as many as 20% of trauma deaths are preventable, with inappropriate prehospital care as one of the largest contributors to preventable trauma deaths.^1–3,11,12 While many studies have characterized incomplete treatment in the hospital setting,^13–15 few studies have characterized treatment appropriateness in the prehospital setting.² Proper characterization of treatment appropriateness in the prehospital setting can facilitate the identification of areas for future training and standardization.^16–18 For example, if airway management is frequently identified as nonindicated, future interventions can potentially lead to development of improved prehospital airway algorithms and integrate additional airway management training into basic life support or advanced life support training and re-certification programs. To better characterize areas for improvement in prehospital care, the Coalition for National Trauma Research is currently conducting the Multi-institutional Multidisciplinary Injury Mortality Investigation in the Civilian Pre-hospital Environment.¹⁹ The goal of this study was to better characterize contributors to preventable prehospital mortality and guide the development of mitigation strategies by analyzing 3,000 prehospital deaths. One limitation of Multi-institutional Multidisciplinary Injury Mortality Investigation in the Civilian Pre-hospital Environment is a reliance on manual case review, which is time-consuming and costly. Development, validation, and utilization of automated processes to determine treatment appropriateness can facilitate rapid characterization of key areas for performance improvement.

Currently, trauma centers rely on time-consuming manual review of trauma reports to determine treatment appropriateness. These reviews are typically done by TMDs and TPMs. Fortunately, settings with missed and nonindicated care are rare; however, this results in the unnecessary manual review of many notes that followed standard of care. An automated system for performance monitoring such as the pilot system described in this study can tag notes with potential gaps in quality to streamline the process of performance review by TPMs and TMDs.

In this sample of 142 patients, IV access was most appropriately delivered with 93.7% of patients receiving an attempt at IV access. Only 69.2% of patients received an airway when it was indicated based on the NLP algorithm developed for this study. There are multiple potential reasons for this finding. First, there are situations where a patient may have indications for an invasive airway to be placed, but EMS providers appropriately managed the patient’s airway using other interventions. In addition, in situations of close proximity to the trauma center, it is our EMS agency’s protocol to use less invasive interventions such as a bag valve mask to ventilate a patient while they transport to the hospital for a more controlled intubation in the trauma bay.²⁰

Surprisingly, few patients with diminished breath sounds and hypotension received needle decompression. In addition, six patients who were hypotensive did not receive TXA or PRBC infusion despite the availability of these resources. Potential reasons include a possibility that this occurred because of proximity to the hospital and subsequent lack of time to perform the intervention before arrival to the emergency department. It is also possible that documented hypotension was transient and resolved before delivery of TXA or PRBC. Finally, we identified a high utilization of crystalloid bolus before or in lieu of PRBC bolus or in patients without hypotension (SBP, >90 mm Hg). Nonindicated fluid management in the prehospital setting has been identified in prior studies as a potential area for future performance improvement in the prehospital setting.^2,21,22

This study suffers from several limitations. First, these results are heavily reliant on documentation practices by EMS providers. However, it is important to clarify that manual chart review by TPM and TMDs is also heavily reliant on documentation practices and the current standard. Given the redundancy of documentation observed in the notes with most patients receiving at least three documented examinations during the course of their EMS transport, we believe this data to be of high quality. Second, this study suffered from poor sample size and only contained records from a single EMS agency. Third, this study excluded patients not admitted to the hospital for trauma and thus by definition did not include patients who died on the scene with a potentially preventable injury (e.g., incomplete management of airway or tension pneumothorax). Fourth, indications were based on locoregional practice. Much debate surrounds the appropriate indications for some treatments in this study. It is important to clarify that the purpose of this study was a pilot to investigate the technological feasibility and accuracy of an NLP pipeline using existing NLP systems to characterize treatment appropriateness as defined by locoregional protocols. Given the high performance of the system, it is feasible that EMS agencies and trauma centers could use this NLP pipeline and modify treatment indication rule sets to align with their locoregional practice. Fifth, the algorithm was limited in its ability to characterize changes in a patient’s condition over time that would have suggested improvement or deterioration of a patient’s condition. Future algorithm optimizations could include temporal analysis of changes in vital signs and clinical characteristics to better classify each patient’s prehospital clinical trajectory. In addition, this study did not evaluate for the presence of an IV before placement of an IO for patients receiving CPR. Thus, it is likely that some patients requiring CPR appropriately did not receive an IO because they already had adequate IV access. Finally, it is possible that certain treatments were delayed when in close proximity to a tertiary trauma center. Unfortunately, our analysis did not include data related to proximity or transport time to a tertiary trauma center, and thus, we were not able to account for this. Future directions will address this limitation of our study by tracking the distance from scene to hospital and evaluate treatments delivered upon arrival in the trauma bay and its association on early clinical outcomes. The results of this study, while preliminary, we believe are compelling with sound technological footing and thus warrant further exploration with external validation.

CONCLUSIONS

In a pilot study of EMS prehospital records, clinical NLP methodologies appear to allow for extraction of procedural indication data and automated timeline summarization. Future directions should focus on optimizing and expanding NLP methods to facilitate performance monitoring.