Natural Language Processing Framework in Early Detection of Amyloidosis

Abstract

Background

Early diagnosis of systemic amyloidosis (SA) is associated with improved outcomes. However, early diagnosis remains limited by variable, nonspecific, and multisystemic presentation.

Objectives

In this study, we used natural language processing (NLP) to develop models for predicting SA diagnosis.

Methods

Patients with a diagnosis of heart failure (HF) and/or neuropathy and sufficient chart data were identified at Oregon Health & Science University (development cohort) and unstructured text data from electronic health records were used to construct NLP-predictive models for the diagnosis of SA. These models were validated in an independent population with HF at Baylor Scott & White Health System (validation cohort).

Results

We identified 109,275 patients with a diagnosis of HF and/or neuropathy (age 57.6 ± 18.6 years, 52.9% females, and 89.6% White) in the development cohort. Of the 420 models evaluated, gradient boosted regression applied to combined notes had the best diagnostic accuracy (area under the curve [AUC] = 0.88), with 73.5% sensitivity, 86.0% specificity, 3.4% positive predictive value, and 99.8% negative predictive value in predicting SA. In the validation cohort (116,658 patients with HF, age 71.6 ± 14.3 years, 47.3% females, 75.8% White), the model had 54.1% sensitivity, 94.0% specificity, 8.8% positive predictive value, and 99.1% negative predictive value, with an AUC of 0.85.

Conclusions

NLP-based prediction models have an excellent diagnostic performance for SA in patients with HF and/or neuropathy. NLP-enabled workflow can potentially lead to earlier diagnosis of amyloidosis, starting therapy and possibly improving outcomes.

Central Illustration

Electronic health records (EHRs) contain a wealth of information that can inform clinical research and decision-making.¹ Detailed and algorithmic evaluation of unstructured clinical narratives allows for the detailed study of patient characteristics without the over-reliance on the International Classification of Diseases (ICD) codes, which are frequently a reflection of physician behavior. Clinical narratives include free-text records of medical events that comprise a wide variety of documentations such as clinical notes from patient visits, diagnostic workups, and laboratory result notes. These narratives are often written in a descriptive way to document information in the EHR; hence, they may contain data not available elsewhere. However, for a rare disease like systemic amyloidosis, the low prevalence further complicates the efforts to screen for the disease. Systemic amyloidosis affects multiple organs and has numerous red flags that are not frequently emphasized aside from being in the clinical notes, such as carpal tunnel syndrome, biceps muscle rupture, tingling and numbness, erectile dysfunction, low voltage electrocardiogram (ECG), among others.²

While utilizing all clinical narrative data from EHR is appealing, it is rather a complex process.² Unstructured, free-text clinical notes take significantly more effort to extract, process, and analyze compared to discrete and coded data. As a result, clinical narratives have not been as extensively leveraged for analyses as structured data.¹ This is even more challenging in rare diseases as the number of diagnosed individuals within a single health care system is small and the clinical narratives have repetitive patterns and text. Natural language processing (NLP) methods have been rapidly developed to allow for the transformation of free text into structured data that can be processed and analyzed.^3,4

Over the past decade, the diagnosis of systemic amyloidosis, especially transthyretin cardiomyopathy (ATTR-CM), has significantly improved due to the introduction of the nonbiopsy diagnosis, improved imaging techniques, as well as availability of novel therapies. However, the ATTR-ACT (Tafamidis Treatment for Patients with Transthyretin Amyloid Cardiomyopathy) study showed better outcomes with tafamidis in patients with earlier disease.⁵

The combination of multisystemic nature of systemic amyloidosis, the many nonspecific red flags, the common presentation of heart failure (HF) or neuropathy, and the availability of therapies make systemic amyloidosis an ideal disease to study using NLP-derived models. To that end, we leveraged EHR-clinical text data from a health care system to develop NLP models for the prediction of systemic amyloidosis and validated those models in an independent health care system. Our secondary objective was to create a general pipeline of NLP-based identification and prediction of rare diseases.

Methods

Development cohort

Study population and data source

The ALARM (nAtural LanguAge pRocessing in aMyloidosis) study is an observational study conducted at Oregon Health & Science University (OHSU). The development cohort was identified as patients evaluated at OHSU between 2008 and 2020 with an ICD version 9th or 10th Revision (ICD-9 or ICD-10) codes of neuropathy, HF, or amyloidosis documented in their EHR. Amyloidosis status was defined based on the presence of ICD-9 (277.3x) or ICD-10 (E85.x) codes in patients’ EHR data. The ICD-9 and ICD-10 codes utilized to identify the study cohort are summarized in Supplemental Table 1.

For patients with a qualifying diagnosis, clinical records were extracted from relational tables in an Oracle database containing data collected through Epic (Epic Systems Corporation) EHR system, then normalized and stored in a research data warehouse. Data included demographics, diagnoses, medications (current, ordered, and administered), ambulatory encounters, hospital encounters, clinical notes, order narratives, laboratory results, pathology reports, and result notes from patients with a qualifying diagnosis in the study period. The study was approved by the OHSU Institutional Review Board. Figure 1 provides an overview of the NLP pipeline.

Figure 1.

Natural Language Processing Modeling Pipeline for Early Detection of Amyloidosis

The flow diagram shows the natural language processing pipeline used in this study. DB = database; LSI = latent semantic indexing; NLP = natural language processing.

Data extraction

For patients with a qualifying diagnosis, patient and visit-level data were extracted using Structured Query Language queries from relational tables in an Oracle database containing data collected through Epic (Epic Systems Corporation) EHR system, then normalized and stored in a research data warehouse. Data included demographics, diagnoses, medications (current, ordered, and administered), ambulatory encounters, hospital encounters, clinical notes, order narratives, laboratory results, pathology reports, and result notes from eligible patients in the study period. These data were stored as rows in comma-separated values files. Each file or series of files, depending on the data type (eg, demographics, clinical notes, diagnoses, etc), contained information from one part of the EHR for patients. Data files were transformed into a Hierarchical Data Format version 5 format using the PyTables Library to allow for efficient processing.^6,7

Data cleaning

Text data were sorted chronologically by date and time, then concatenated into a single-level document. Cleaning text documents included converting text to all lowercase, removing short (<3 characters) and long (>64 characters) sets of continuous nonspace characters, and substituting all nonletter characters with space characters. For each data type, a single column of text data was selected as the source of clinical narrative information. A deduplication algorithm was implemented to eliminate repeated text for each type of clinical narrative for individual patients. The algorithm searches through text and records all unique sets of 30 characters in a row. If a duplicate set of 30 characters is identified, then it would be marked for removal.

Tokenization

Subsequently, raw text was converted into a set of words, or tokens, which were listed in a dictionary and formed a bag-of-words. Tokenization and the creation of dictionaries occurred for each type of clinical narrative used. Words that occurred with high frequency (those occurring in >60% of documents) across the entire data set were excluded as were those that were common in patients with an amyloidosis diagnosis code (occurring >85%) or infrequent in the data set (occurring <10% in patients with amyloidosis and <30% without the disease). To mitigate clues for the algorithm that we were not be able to account for, an OR was computed to compare word occurrence among patients with an amyloidosis ICD code (“cases”) and the same word occurrence in those without an amyloidosis ICD code (“noncases”):

$$OR=\left[\frac{\left(\frac{\#caseswithword}{\#ofcaseswithoutword}\right)}{\left(\frac{\#non-caseswithword}{\#ofnon-caseswithoutword}\right)}\right]$$

This approach allowed for the exclusion of words that were disproportionally more frequent in cases than in noncases and cannot be easily controlled for—such as “amyloidosis,” “Tafamidis,” or the name of a physician who primarily treats amyloidosis (Supplemental Table 2). Importantly, the word frequency and OR exclusions were only computed on the training data set, without input from the test data set. A range of OR between 1 and 50 were tested, and an OR of 3.5 was selected. A 50,000-word dictionary was created from the 10,000 and 40,000 most common nonexcluded words from cases and noncases, respectively. We applied Term Frequency-Inverse Document Frequency weighting to the bag-of-words created from the dictionary to further normalize them. The resulting matrix was transformed using latent Semantic Indexing (LSI) for topic modeling. LSI matrices with 50, 100, and 200 dimensions were created.

Model development and training

Patients were randomly partitioned into training (80% of cohort) and testing (20% of cohort) data sets. All data from the testing set was held out during training and used only when assessing model performance. Using Scikit-learn package in Python, several predictive models for identifying amyloidosis status using the LSI vectors from clinical notes were developed including gradient boosted logistic regression (XGBoost),⁸ locally weighted regression using K-nearest neighbor, logistic regression, random forest, and support vector machines.⁹ Receiver operator characteristic (ROC) curve analysis was used to evaluate the quality of models and the cutoff for classifying patients into cases and noncases was selected by maximizing Youden’s index (sensitivity + specificity − 1). We followed an iterative strategy for selecting parameters and modeling approach with data from the training set using sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) based on the cutoff as well as area under the ROC curve (AUC) to evaluate the models. Among the methods assessed, XGBoost had the highest performance for the majority of models built on the training set and was selected as the modeling approach.⁸ A number of model parameters were varied in the XGBoost models to explore which combination would produce the most predictive model. Models were run for the following parameters, totaling 420 different combinations: 1) type of clinical data (clinical note, diagnosis descriptions, lab result, results notes, and a combination of these notes; 2) OR for word exclusion (1.5, 2, 3.5, 5, 10, 20, and 50); 3) LSI vector dimensions (50, 100, and 200); and iv) XGBoost Scale Weight (10, 100, 200, and 400). Once parameters were selected to optimize model performance, they were used in final models built on the training data for each note type then applied to the test data set. We reported sensitivity, specificity, PPV, and NPV as well as area under the curve (AUC) from ROC curves from these scenarios with corresponding 95% CIs based on 5,000 bootstrap replicates. Sensitivity analyses were conducted, repeating the described approach with a subset of data restricted to: 1) clinical notes from 2010 and after (when Epic had been consistently implemented throughout our health care system); 2) 3 or more notes per patient; and 3) clinical notes present for >180 days per patient. Data preprocessing, NLP analyses, and initial parameter testing were conducted in Python 3.7 Anaconda (Anaconda Software Distribution, Anaconda Inc) and analyses of final models were conducted in R version 4.3.1 (R Core Team) using the caret, pROC, and XGBoost packages.^8,10, 11, 12 The code for preprocessing NLP is deposited on GitHub (https://github.com/MasriAhmadMD/ALARM_Study, GitHub, Inc) with an open-access license.

External validation cohort

Patients with HF and/or amyloidosis diagnoses in the Baylor Scott & White (BSW) Epic EHR between April 1, 2020, and March 31, 2023, were included in the validation cohort. Patients <18 years, heart transplant, and left ventricular assist device recipients were excluded. The algorithm developed at OHSU was independently validated and tested on clinical notes (including progress notes, chart notes, provider notes, procedure notes, and lab notes) from BSW’s EHR database (Supplemental Figure 6). This validation aimed to assess the model’s performance in a new setting and establish its robustness and adaptability. Structured Query Language scripts were also provided by OHSU to replicate their data extraction process, which enabled the BSW team to generate a comparable data set from BSW’s EHR. The study was approved by the BSW Institutional Review Board.

This data set was structured to align with the data types and attributes used in the development process. To ensure accurate replication, the original code was provided. The repository contained code for critical components of the NLP pipeline, with essential steps involving Hierarchical Data Format version 5 text compression, deduplication, data cleaning, and time-space sorting. To minimize this heterogeneity, similar processes were employed, including similar preprocessing (standardization, tokenization, deduplication) and NLP transformation pipeline (topic modeling, latent semantic indexing) while evaluated the OHSU's original locked model with the fixed threshold at BSW (no retraining or tuning). Following data preprocessing, the NLP pipeline replicated the development approach, including the bag-of-words method, the generation of a Term Frequency-Inverse Document Frequency matrix, and topic modeling with LSI to create feature vectors. Furthermore, the exact Python 3.7 environment was replicated for the machine learning (ML) components by providing a requirements file from their GitHub repository. This ensured that all dependencies and libraries matched those used in the original model, supporting consistency in model behavior and output. Subsequently, serialized Python pickle files were provided by the OHSU team, which included parameters from the final model and corresponding LSI feature weights. The BSW team tested these in their environment, applying the model to the replicated data set. This process enabled the team to generate predictions and assess the model's performance on BSW’s data, validating the model’s applicability, accuracy, and effectiveness within their own organization.

Results

Between January 1, 2008, and December 31, 2020, we identified 109,275 patients with a qualifying diagnosis of HF and/or neuropathy, and of those, 83,297 patients (n = 66,535 in the training set, n = 16,762 in the testing set) were included in the study cohort. Table 1 summarizes the demographic characteristics of the patients in the cohort. The mean age of patients was 57.6 ± 18.6 years, 44,047 (52.9%) were female, and 74,626 (89.6%) were White. At the time of model creation, a total of 583 patients (472 in training set and 111 in validation set) or 0.7% of the study cohort was found to have a diagnosis of amyloidosis based on ICD-9 and ICD-10 diagnostic codes. Framework utilizing NLP for early detection of amyloidosis is summarized in Central Illustration.

Table 1.

Patient Characteristics by Inclusion Diagnosis and Test Data Set Status^a

	Training Data Set			Test Data Set
	Neuropathy (n = 44,112)	Heart Failure (n = 27,877)	Neuropathy/Heart Failure (n = 66,535)	Neuropathy (n = 11,179)	Heart Failure (n = 6,930)	Neuropathy/Heart Failure (n = 16,762)
Female, n (%)	25,358 (57.5)	12,442 (44.6)	35,240 (53.0)	6,416 (57.4)	3,043 (43.9)	8,807 (52.5)
Age, y, mean (SD)b	54.6 (16.9)	63.5 (19.6)	57.6 (18.6)	54.8 (17.0)	63.6 (19.2)	57.6 (18.7)
Race, n (%)
White	39,672 (89.9)	24,904 (89.3)	59,631 (89.6)	10,034 (89.8)	6,193 (89.3)	14,995 (89.5)
Black	1,098 (2.0)	821 (2.9)	1725 (2.6)	288 (2.5)	206 (3.0)	450 (2.7)
Asian/Pacific Islander	1,015 (2.3)	586 (2.1)	1,500 (2.3)	294 (2.6)	146 (2.1)	418 (2.5)
Other/unknown	2,327 (5.3)	1,566 (5.6)	3,679 (5.5)	563 (5.0)	385 (5.6)	899 (5.4)
Hispanic, n (%)	2012 (4.6)	1,055 (3.8)	2,899 (4.4)	491 (4.4)	265 (3.8)	708 (4.2)
Amyloidosis diagnosis, n (%)c	256 (0.6)	347 (1.2)	472 (0.7)	70 (0.6)	76 (1.1)	111 (0.7)

EHR = electronic health record; ICD = International Classification of Diseases.

^aPatients overlap in diagnosis categories.

^bAge at first qualifying diagnosis for inclusion in cohort.

^cAmyloidosis diagnosis at some point in EHR based on ICD-9 or ICD-10 diagnoses codes.

Central Illustration.

Framework Utilizing Natural Language Processing for Early Detection of Amyloidosis

∗Intended as upstream screening aid to prompt evaluation; not a standalone diagnostic. AUC = area under the curve; BSW = Baylor Scott & White; EHR = electronic health record; NPV = negative predictive value; OHSU = Oregon Health & Science University; PPV = positive predictive value; XGBoost = gradient boosted regression; other abbreviation as in Figure 1.

Model development and selection

A total of 420 XGBoost models were evaluated, and the top performing 30 models are shown in Supplemental Table 3. The best performing model had a Youden’s index of J = 0.4131 and the worst performing model had J = 0.0082. The 420 models evaluated had an average Youden’s index of J = 0.2232 with an SD of 0.0855. The best performing models contained data from either “clinical notes” only or “combination” of multiple types of clinical text, suggesting that clinical notes were sufficient and yielded similar results. In addition, the highest scale weight of 400 was common in models with the highest Youden’s index. Using a scale weight slightly larger than the LSI vector dimensions appeared to produce the best results for these models. Finally, higher OR for word exclusion led to improved prediction, but models with an OR of 5 or greater typically included “keywords” that were indicative of the patient already having been worked up for amyloidosis. Therefore, for predictive modeling, an OR of 3.5 or less was used despite the drop in performance. After preliminary model building, we used XGBoost to build the final models on training data using a scale weight of 400, 100 LSI vectors, and an OR of 3.5.

Model performance in overall cohort (neuropathy and heat failure patients)

Table 2 summarizes the performance of the selected models built with XGBoost to identify amyloidosis in various narrative types (eg, clinical notes, diagnostic notes, lab notes, and lab results). A model based on “combined notes” had 73.5% (95% CI: 65.5%-81.4%) sensitivity, 85.6% (95% CI: 85.1%-86.1%) specificity, 3.4% (95% CI: 3.0%-3.7%) PPV, 99.8% (95% CI: 99.7%-99.9%) NPV, and an AUC of 0.88 (95% CI: 0.84-0.91). Similarly, a model based on “clinical notes” had 78.8% (95% CI: 70.8%-85.8%) sensitivity, 81.8% (95% CI: 81.2%-82.4%) specificity, 2.9% (95% CI: 2.6%-3.1%) PPV, 99.82% (95% CI: 99.76%-99.88%) NPV, and an AUC of 0.87 (95% CI: 0.84-0.90). Models based on “lab result notes” had the lowest diagnostic performance (AUC = 0.80; 95% CI: 0.75-0.84) in predicting amyloidosis. The ROC curve of the model in various narrative types is shown in Figure 2A.

Table 2.

NLP Predictive Models

	Sensitivity (95% CI)a	Specificity (95% CI)a	PPV (95% CI)a	NPV (95% CI)a	AUC (95% CI)a
Neuropathy patients
Clinical notes	0.764 (0.667-0.861)	0.832 (0.825-0.838)	0.029 (0.025-0.032)	0.998 (0.997-0.999)	0.857 (0.811-0.901)
Diagnosis notes	0.694 (0.583-0.792)	0.775 (0.768-0.783)	0.020 (0.017-0.023)	0.997 (0.997-0.998)	0.817 (0.768-0.863)
Lab notes	0.681 (0.569-0.778)	0.779 (0.772-0.787)	0.020 (0.016-0.023)	0.997 (0.996-0.998)	0.807 (0.752-0.859)
Lab result notes	0.722 (0.611-0.819)	0.682 (0.673-0.690)	0.015 (0.012-0.017)	0.997 (0.996-0.998)	0.775 (0.716-0.830)
Combined notes	0.444 (0.333-0.556)	0.939 (0.933-0.942)	0.045 (0.033-0.056)	0.996 (0.995-0.997)	0.855 (0.820-0.892)
Heart failure patients
Clinical notes	0.829 (0.737-0.908)	0.780 (0.770-0.789)	0.040 (0.036-0.044)	0.998 (0.996-0.999)	0.856 (0.800-0.902)
Diagnosis notes	0.763 (0.671-0.855)	0.712 (0.700-0.722)	0.029 (0.025-0.032)	0.996 (0.995-0.998)	0.819 (0.765-0.867)
Lab notes	0.724 (0.618, 0.816)	0.829 (0.819-0.837)	0.045 (0.038-0.051)	0.996 (0.995-0.998)	0.860 (0.811-0.903)
Lab result notes	0.553 (0.434-0.658)	0.839 (0.830-0.848)	0.037 (0.029-0.044)	0.994 (0.993-0.996)	0.762 (0.697-0.823)
Combined notes	0.592 (0.487-0.697)	0.934 (0.928-0.940)	0.090 (0.073-0.108)	0.995 (0.994-0.996)	0.866 (0.817-0.909)
Neuropathy or heart failure patients
Clinical notes	0.788 (0.708-0.858)	0.818 (0.812-0.824)	0.028 (0.026-0.031)	0.998 (0.998-0.999)	0.874 (0.841-0.904)
Diagnosis notes	0.708 (0.619-0.788)	0.816 (0.810-0.822)	0.025 (0.022-0.028)	0.998 (0.997-0.998)	0.831 (0.788-0.870)
Lab notes	0.628 (0.540-0.717)	0.832 (0.826-0.838)	0.025 (0.021-0.028)	0.997 (0.996-0.998)	0.827 (0.786-0.865)
Lab result notes	0.673 (0.584-0.761)	0.779 (0.773-0.786)	0.020 (0.018-0.023)	0.997 (0.996-0.998)	0.797 (0.750-0.844)
Combined notes	0.735 (0.655-0.814)	0.856 (0.851-0.861)	0.034 (0.030-0.037)	0.998 (0.997-0.999)	0.878 (0.842-0.911)

AUC = area under the curve; NPV = negative predictive value; NLP = natural language processing; PPV = positive predictive value; XGBoost = gradient boosted regression.

Performance of predictive models built with XGBoost in identifying amyloidosis by patient diagnosis and narrative type.^b

^aValues based on the cut point selected by maximizing Youden’s index in the training data set.

^bLogistic gradient boosting models fit to training data set then applied to test data. Preprocessed and cleaned narrative data for each narrative type contains a 100-dimensional LSI matrix that was constructed with NLP analyses using an OR of 3.5 for word exclusion; the combined notes contain the LSI matrices for each component narrative type (ie, 4 100-dimensional LSI matrices).

Figure 2.

Model’s Receiver Operator Characteristic Curve

(A) overall cohort (neuropathy and heart failure), (B) patients with neuropathy, and (C) patients with heart failure. ∗Area under the receiver operator characteristic curve for each narrative type: (A) chart notes: 0.87 (95% CI: 0.84-0.90), diagnosis descriptions: 0.83 (95% CI: 0.79-0.87), lab notes: 0.83 (95% CI: 0.79-0.87), lab result notes: 0.80 (95% CI: 0.75-0.84), combined notes: 0.88 (95% CI: 0.84-0.91); (B) chart notes: 0.86 (95% CI: 0.80-0.90), diagnosis descriptions: 0.82 (95% CI: 0.77-0.87), lab notes: 0.86 (95% CI: 0.81-0.90), lab result notes: 0.76 (95% CI: 0.70-0.82), combined notes: 0.87 (95% CI: 0.82-0.91); (C) chart notes: 0.86 (95% CI: 0.81-0.90), diagnosis descriptions: 0.82 (95% CI: 0.77-0.86), lab notes: 0.81 (95% CI: 0.75-0.86), lab result notes: 0.78 (95% CI: 0.72-0.83), combined notes: 0.86 (95% CI: 0.82-0.89).

Model performance in neuropathy patients

A total of 326 patients (256 in training set and 70 in testing set) with neuropathy had an amyloidosis diagnosis. The performance of the selected models built with XGBoost in identifying amyloidosis in patients with neuropathy (n = 55,291) is summarized in Table 2. The NLP model based on “combined notes” had 44.4% (95% CI: 33.3%-55.6%) sensitivity, 93.9% (95% CI: 93.3%-94.2%) specificity, 4.5% (95% CI: 3.3%-5.6%) PPV, 99.6% (95% CI: 99.5%-99.7%) NPV, with an AUC of 0.86 (95% CI: 0.82-0.89). On the other hand, an NLP model based on “clinical notes” only had 76% (95% CI: 66.7%-86.1%) sensitivity, 83% (95% CI: 82.5%-83.8%) specificity, 2.9% (95% CI: 2.5%-3.2%) PPV, and 99.8% (95% CI: 99.7%-99.9%) NPV, and an AUC of 0.86 (95% CI: 0.81-0.90). The ROC curve of the model for neuropathy patients in various narrative types is shown in Figure 2B.

Model performance in heart failure patients

A total of 423 patients (347 in training set and 76 in testing set) with HF had an amyloidosis diagnosis. The performance of the selected models in identifying amyloidosis in HF patients (n = 34,807) is shown in Table 2. For the prediction of amyloidosis, an NLP model based on “combined notes” had 59.2% (95% CI: 48.7%-69.7%) sensitivity, 93.4% (95% CI: 92.8%-94.0%) specificity, 9.0% (95% CI: 7.3%-10.8%) PPV, 99.5% (95% CI: 99.4%-99.6%) NPV, with an AUC of 0.87 (95% CI: 0.82-0.91). While an NLP model based on “clinical notes” only had 82.9% (95% CI: 73.7%-90.8%) sensitivity, 78% (95% CI: 77.0%-78.9%) specificity, 4.0% (95% CI: 3.6%-4.4%) PPV, 99.8% (95% CI: 99.6%-99.9%) NPV, and an AUC of 0.86 (95% CI: 0.80-0.90). The ROC curve of the model in various narrative types is shown in Figure 2C.

Sensitivity analysis using restricted data sets

Results for sensitivity analyses are shown in Supplemental Table 4. In summary, the models’ performance in neuropathy patients, HF patients, and the overall cohort was similar (AUC of 0.83 [95% CI: 0.78-0.88], 0.84 [95% CI: 0.78-0.89], and 0.86 [95% CI: 0.82-0.90] respectively, for combined notes) to that of the full data set. The ROC curves for neuropathy, HF, and neuropathy/HF patients with restricted data set are shown in Supplemental Figures 7-9.

External validation

A total of 116, 658 patients with HF were included in the validation cohort. The mean age was 71.6 ± 14.3 years, 55,276 (47.3%) were females, and 88,533 (75.8%) were White. A total of 1,234 patients (1.1%) had a diagnosis of systemic amyloidosis. Using the combined clinical notes from the HF cohort, the algorithm had 54.1% (95% CI: 44.3%-55.2%) sensitivity, 94.0% (95% CI: 93.6%-94.3%) specificity, 8.8% (95% CI: 8.3%-10.2%) PPV, 99.1% (95% CI: 99.0%-99.2%) NPV, with an AUC of 0.85 (95% CI: 0.83-0.87).

Discussion

In this study of >200,000 patients across 2 health care systems, an NLP-based algorithm applied to chart notes had excellent performance in predicting systemic amyloidosis. In the development cohort, the AUC was 0.88 with 74% sensitivity, 87% specificity, 3.4% PPV, and 99% NPV in predicting systemic amyloidosis in patients with neuropathy or HF. Similar diagnostic performance was noted in patients with HF except for a lower sensitivity. PPV was improved by utilizing combined notes across the EHR. In addition, the algorithm had a similar performance in the validation HF cohort in an independent health care system. Collectively, these findings provide an open-source NLP algorithm to screen for systemic amyloidosis which can be adapted at the hospital level, show the feasibility and excellent performance of such an algorithm in a rare disease with low overall prevalence like systemic amyloidosis.

Systemic amyloidosis, in particular ATTR-CM, has become a commonly encountered disease in patients with HF. However, ATTR-CM remains a grossly underdiagnosed condition given due to lack of systematic screening strategies in populations at risk.¹³ The improved noninvasive diagnosis of ATTR-CM still did not enable a significant shift in identifying patients with very early disease.¹⁴ With the increased clinical recognition, therapies have emerged to stabilize the disease trajectory but do not improve cardiac function. In pivotal clinical trials of stabilizers and silencers, earlier institution of therapy is associated with better outcomes.^5,15,16 Similarly, in patients with ATTR polyneuropathy, earlier institution of therapy stabilizes the disease but does not significantly improve nerve function.^17,18 As such, earlier identification of ATTR-CM and ATTR polyneuropathy to enable earlier institution of therapy promises to be the best approach of improving patient outcomes. Similarly, light chain amyloidosis represents a medical emergency when it presents with significant HF. The nonspecific nature of the presentation combined with the low incidence represents a major challenge in the timely identification of this important medical condition. Patients’ outcomes and experiences are fully driven by the degree of cardiac involvement.¹⁹ As such, strategies that enable earlier identification of light chain amyloidosis will lead to improved survival.

The nonspecific nature of red flags for systemic amyloidosis requires a strategy that does not rely on the coding behavior of physicians and other health care providers. While ICD codes are useful and can be leveraged for screening for systemic amyloidosis,²⁰ they rely on local practices and behavior and are unlikely to widely capture these red flags. Red flags such as orthopedic conditions, autonomic symptoms, and weight loss are unlikely to be coded using ICD codes and that is the power of NLP, where one relies on any documentation in the chart to raise suspicion. Furthermore, the use of OR represents a scalable approach that allows for better application of this algorithm across different health care systems with distinct behaviors and practices.

Developing artificial intelligence (AI)-driven tools for early detection of amyloidosis has garnered considerable attention to counter the diagnostic challenges of systemic amyloidosis. Several studies have demonstrated the effectiveness of AI models applied to specific types of data—including laboratory results, ECGs, transthoracic echocardiograms, bone scintigraphy, and cardiac magnetic resonance imaging (MRI)—for automated identification of ATTR in HF populations.21, 22, 23, 24 These approaches leverage the simultaneous analysis of multiple features that reflect early physiological changes and subtle indicators of cardiac involvement. In contrast, NLP algorithms applied to medical records identify patients with a higher likelihood of systemic amyloidosis by detecting a pattern of nonspecific but clinically relevant systemic features—such as neuropathy, carpal tunnel syndrome, or unexplained weight loss—documented in unstructured clinical notes. As such, the current NLP-based algorithm is meant to be applied upstream of the diagnostic algorithm.

Huda et al validated an ML model based on administrative medical claims in EHR of 1,071 patients with ATTR amyloidosis and 1,071 HF patients without amyloidosis.²⁵ Their random forest models leveraged ORs of both cardiac and noncardiac clinical phenotypes predictive of ATTR and were tested across 4 validation cohorts, yielding AUC values ranging from 0.76 to 0.95. The investigators evaluated 3 supervised ML algorithms, in contrasted to the approaches employed in our study. Although supervised algorithms can achieve high accuracy, they often struggle to generalize beyond the scope of their training data and may perform poorly in novel or unstructured scenarios.^26,27 In contrast, unsupervised models are better suited to uncovering broader, latent patterns not limited by predefined labels. Moreover, administrative claims data only capture specific portions of the medical record and suffer from frequent missingness which at times limits generalizability. In the current study, external validation in an independent health care system was meant to overcome the generalizability challenge.

Uncommon diseases represent a particular challenge for screening algorithms. The relatively low prevalence across a whole population leads to a high number of false positive cases, leading to the low PPV. This is a limitation that applies to conditions like systemic amyloidosis and others. However, the goal of such an NLP algorithm is to be a nudge to the physician or health care provider, allowing them to put together a set of red flags that are hard to manually synthesize. The input from the clinician, which cannot be quantified, will significantly limit the false positive cases. As such, the goal of this NLP algorithm is not to achieve diagnostic abilities for systemic amyloidosis, but rather, raise the suspicion in an attempt to limit under- and delayed-diagnosis.

Study limitations

The clearest limitation in developing an NLP-based model in amyloidosis is the rare nature of the disease and the narrow scope of care that is delivered to these patients by specific individuals within each health care system. Such practices introduce certain signatures in the data sets that have nothing to do with the clinical presentation. Because of this, algorithm development requires significant reiterative processes to ensure that nonclinical data are not being used to select patients with amyloidosis. In addition, the rare disease nature does affect the predictive value of any model, driven by disease prevalence. The improved recognition of systemic amyloidosis coupled with democratizing the care of such patients even in community-based settings will improve the performance of such models. We have a limited ability to gain insights toward why a certain set of words that are not necessarily informative to us lead to the detection of amyloidosis. All these limitations drove the reason for the independent validation that was done at an external health care system. While the model was developed at a tertiary health care system, it was validated at a multisite large academic and community-based health care system.

In our study, the goal was to identify systemic amyloidosis, and we were underpowered due to small sample size from further subgrouping amyloidosis into multiple different entities. In addition, the ICD codes utilized to identify study cohort are generally nonspecific and should not be used to identify specific amyloidosis subtypes. Additionally, relying solely on ICD codes for case-control classification is prone for coding inaccuracies.

In addition, our analysis aimed to assess the diagnostic performance and robustness of NLP-based prediction models for amyloidosis, and as such, did not include comparator groups. Future research is warranted to determine the net clinical benefit of large-scale implementation of such models within health care systems—for instance, by comparing their performance against established clinical risk scores or through decision curve analysis. Furthermore, independent validation of NLP models in non-HF populations, such as cohorts with peripheral neuropathy, would strengthen the model’s early detection claims and help clarify its potential role in broader clinical screening. Finally, the model will require adaptation to health care systems that use distinct EHR data structures, such as structured data elements only.

Conclusions

NLP-based prediction models have an excellent diagnostic performance for systemic amyloidosis in patients with HF and neuropathy. The performance of the NLP algorithm was validated in an external health care system and maintained its diagnostic accuracy. NLP-enabled workflow can potentially lead to earlier diagnosis of systemic amyloidosis, starting therapy and possibly improving outcomes.

Funding support and author disclosures

The study was funded by Ionis and AstraZeneca. Dr Butler reports consultant to Abbott, American Regent, Amgen, Applied Therapeutic, AskBio, Astellas, AstraZeneca, Bayer, Boehringer Ingelheim, Boston Scientific, Bristol Myers Squibb, Cardiac Dimension, Cardiocell, Cardior, CSL Bearing, CVRx, Cytokinetics, Daxor, Edwards, Element Science, Faraday, Foundry, G3P, Innolife, Impulse Dynamics, Imbria, Inventiva, Ionis, Lexicon, Lilly, LivaNova, Janssen, Medtronics, Merck, Occlutech, Owkin, Novartis, Novo Nordisk, Pfizer, Pharmacosmos, Pharmain, Prolaio, Regeneron, Renibus, Roche, Salamandra, Sanofi, SC Pharma, Secretome, Sequana, SQ Innovation, Tenex, Tricog, Ultromics, Vifor, and Zoll. Dr Priest has received research support from AstraZeneca, Owkin, Boehringer Ingelheim, and CSL Vifor. Dr Masri has received consultant/advisor fees from Tenaya, Attralus, Cytokinetics, Bristol Myers Squibb, Eidos, Pfizer, Haya, Lexion, BioMarin, Alexion, and Ionis, and research grants from Ionis, Pfizer, Cytokinetics, and, Attralus. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.