A Methodology to Generate Longitudinally Updated Acute‐On‐Chronic Liver Failure Prognostication Scores From Electronic Health Record Data

Abstract

Queries of electronic health record (EHR) data repositories allow for automated data collection. These techniques have not been used in hepatology due to the inability to capture hepatic encephalopathy (HE) grades, which are inputs for acute‐on‐chronic liver failure (ACLF) models. Here, we describe a methodology to use EHR data to calculate rolling ACLF scores. We examined 239 patient admissions with end‐stage liver disease from July 2014 to June 2019. We mapped EHR flowsheet data to determine HE grades and calculated two longitudinally updated ACLF scores. We validated HE grades and ACLF diagnoses by chart review and calculated sensitivity, specificity, and Cohen’s kappa. Of 239 patient admissions analyzed, 37% were women, 46% were non‐Hispanic white, median age was 60 years, and the median Model for End‐Stage Liver Disease–Na score at admission was 25. Of the 239, 7% were diagnosed with ACLF as defined by the North American Consortium for the Study of End‐Stage Liver Disease (NACSELD) diagnostic criteria at admission, 27% during the hospitalization, and 9% at discharge. Forty percent were diagnosed with ACLF by the European Association for the Study of the Liver– Chronic Liver Failure Consortium (CLIF‐C) diagnostic criteria at admission, 51% during the hospitalization, and 34% at discharge. From the chart review of 51 admissions, we found sensitivities and specificities for any HE (grades 1‐4) were 92%‐97% and 76%‐95%, respectively; for severe HE (grades 3‐4), sensitivities and specificities were 100% and 78%‐98%, respectively. Cohen’s kappa between flowsheet and chart review of HE grades ranged from 0.55 to 0.72. Sensitivities and specificities for NACSELD‐ACLF diagnoses were 75%‐100% and 96%‐100%, respectively; for CLIF‐C‐ACLF diagnoses, these were 91%‐100% and 96‐100%, respectively. We generated approximately 28 unique ACLF scores per patient per admission day. Conclusion: We developed an informatics‐based methodology to calculate longitudinally updated ACLF scores. This opens new analytic potentials, such as big data methods, to develop electronic phenotypes for patients with ACLF.

Queries of electronic health record (EHR) data repositories allow for automated data collection. These techniques have not been utilized in hepatology due to previous inability to capture hepatic encephalopathy (HE) grades, which are inputs for acute‐on‐chronic liver failure (ACLF) models. We describe a methodology to utilizing EHR data to calculate rolling ACLF scores.

Abbreviations

ACLF

acute‐on‐chronic liver failure

APASL

Asian Pacific Association for the Study of the Liver

CDR

clinical data repository

CI

confidence interval

CLIF

chronic liver failure

CLIF‐C

Chronic Liver Failure Consortium

EASL

European Association for the Study of the Liver

EASL

European Association for the Study of the Liver

EHR

electronic health record

EPIC

EpicCare

ESLD

end‐stage liver disease

FiO₂

oxygen fraction

FrAILT

Multi‐Center Functional Assessment in Liver Transplantation Study

GCS

Glasgow Coma Score

HE

hepatic encephalopathy

IQR

interquartile range

LFI

Liver Frailty Index

MELD

Model for End‐Stage Liver Disease

MELD‐Na

Model for End‐Stage Liver Disease–Sodium

NACSELD

North American Consortium for the Study of End‐Stage Liver Disease

SQL

Structured Query Language

WHC

West Haven Criteria

Electronic health records (EHRs) capture and generate vast amounts of granular clinical data through routine operations.^{( 1 )} Structured Query Language (SQL) queries of associated clinical data repositories (CDRs) allow for automated generation of comprehensive laboratory, flowsheet, medical device, and medication administration reports for a cohort of patients.^{( 2} , ³ , ^{4 )} Integration of these separate data reports have the potential to survey patients in a longitudinal fashion during inpatient admission and construct electronic phenotypes to define subgroups of interest for further exploration.^{( 5} , ^{6 )} Existing applications of SQL querying of data repositories in gastroenterology and hepatology, however, have been limited to searching International Classification of Diseases (ICD) 9/10 codes, identifying keywords in clinician documentation, and/or acquiring laboratory data.^{( 7} , ⁸ , ^{9 )}

In hepatology research specifically, the adoption of informatic methods described above has been hindered by the inability to capture data to inform hepatic encephalopathy (HE) grades, which are often used as inputs into clinical prognostication models. EHR flowsheets, which contain structured and semistructured entries reflecting interprofessional assessments of mentation, functional status, and physical exam findings, represent a rich source of relevant clinical information.^{( 3} , ^{4 )} These semistructured documentations of mentation have the potential to be mapped to describe HE, thereby enabling en masse automated data acquisition for clinical research in hepatology.

This becomes especially relevant in the study of acute‐on‐chronic liver failure (ACLF), which is defined as the acute decompensation of end‐stage liver disease (ESLD) with extrahepatic organ failures and high short‐term mortality.^{( 10} , ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵ , ^{16 )} ACLF is a heterogeneous and dynamic clinical syndrome with variable etiologies, triggers, and outcomes.^{( 17 )} Reflecting the diversity of ACLF, several competing definitions and scoring systems currently exist, such as the North American Consortium for the Study of End‐Stage Liver Disease (NACSELD) diagnostic criteria and NACSELD‐ACLF score^{( 10 )}, the European Association for the Study of the Liver– Chronic Liver Failure (EASL‐CLIF) diagnostic criteria and CLIF‐Consortium‐ACLF (CLIF‐C‐ACLF) score,^{( 13 )} and the Asian Pacific Association for the Study of the Liver (APASL) ACLF Research Consortium definition and score.^{( 16 )} Notably, the APASL ACLF definition differs in comparison to European (EASL‐CLIF) and North American (NACSELD) definitions as it includes chronic liver diseases without cirrhosis and only considers precipitants that are intrahepatic in origin.^{( 17} , ^{18 )}

Despite these differences in definitions of ACLF, the existing ACLF prognostication scores are generated in a cross‐sectional manner at a specific point in time and remain limited in clinical utility due to inconsistent abilities to predict recovery and identify transplant candidates.^{( 19 )} Moreover, the lack of consensus on a unified prognostication model implies that current methodologies for predictive modeling may be inadequate for this disease state. Longitudinally updated ACLF scores, therefore, may be able to improve predictive ability and better inform ACLF outcomes research as studies have shown score changes and trajectories have greater prognostic value.^{( 20} , ²¹ , ²² , ^{23 )}

In this study, we describe a two‐step methodology to generate longitudinally updated ACLF prognostication scores applicable to patients with ESLD in the United States (NACSELD‐ACLF and CLIF‐C‐ACLF scores):

1. Calculate West Haven Criteria (WHC) grades of HE by mapping mentation and functional status descriptors in flowsheet reports and validating these mapped WHC grades through chart review.

2. Integrate mapped WHC grades with relationally linked reports of laboratory value, medical device data, and medication administration reports to generate longitudinally updated ACLF prognostication scores.

Materials and Methods

We examined all inpatient admissions in a 5‐year period from July 1, 2014, through June 30, 2019, for the 1,918 patients enrolled in the Multi‐Center Functional Assessment in Liver Transplantation (FrAILT) Study at a single academic medical center (University of California, San Francisco Medical Center) as of October 30, 2019. The FrAILT Study is a prospective longitudinal study of adult patients with ESLD awaiting liver transplantation and evaluated in the ambulatory care setting.^{( 24 )} Hospital admissions for these patients were excluded if the admission took place after liver transplantation, was for a scheduled liver transplantation within 48 hours, hospital stays were ≤24 hours, or if transplantation took place before their enrollment in the FrAILT Study. If a patient had multiple hospitalizations, we analyzed the hospitalization immediately after the most recent Liver Frailty Index (LFI) assessment to isolate one admission per patient (patient admission). Of note, all patients who are listed for liver transplantation at our medical center are admitted to a dedicated multidisciplinary Liver Transplant Unit jointly attended by a hepatologist and a transplant surgeon for inpatient care. A flow diagram of the analyzed patient population from the FrAILT Study is shown in Fig. 1.

FIG. 1.

Isolation of the 239 patient admissions analyzed in this study.

Baseline demographic and clinical data were extracted from the date of the latest outpatient LFI assessment. Race/ethnicity was classified into the following categories: white, black, Hispanic, Asian, Native American, or other. Etiologies of liver disease were categorized as: chronic hepatitis C, alcohol‐associated, autoimmune/cholestatic, chronic hepatitis B, and other etiologies. Patients were considered to comorbid diagnoses of hypertension, diabetes, or coronary artery disease if they were reported in the EHR. The Institutional Review Board at the University of California, San Francisco, approved this study.

SQL Data Collection

For this cohort of 1,918 patients and eligible admissions, we queried the EpicCare (EPIC) (Epic Systems, Verona, WI) Clarity CDR hosted at the University of California, San Francisco Medical Center for reports containing data generated through routine care (Table 1). These reports were then linked relationally through two unique identifiers, medical record number and contact serial number, to each patient and admission.

TABLE 1.

Data and Report Elements Generated from SQL Queries of Clarity CDR

Clarity CDR Report Elements
Flowsheet report	‐ Structured documentation of speech, cognition, orientation level, level of consciousness, and GCS in nursing flowsheets
	‐ Oxygen device, peripheral capillary oxygen saturation measurements, and blood pressures by peripheral and arterial measurements in vital signs flowsheets
laboratory data report	All laboratory (such as complete blood count, basic metabolic panel, liver function tests, coagulation parameters, albumin, and others) and bacterial culture (such as urine, central blood, peripheral blood, peritoneal fluid, sputum, and others) orders and results
Dialysis order report	All provider orders for hemodialysis, continuous venovenous hemofiltration, continuous venovenous hemodialysis, peritoneal dialysis, and ultrafiltration
Ventilator data report	Mechanical ventilation use (bilevel positive airway pressure or mechanical ventilator), ventilator mode (such as pressure support, assist control, and others), fraction of inspired oxygen (FiO2), and partial pressure to fraction of oxygen ratio, if available
Vasopressor administration report	Medication administration records and times for all administrations of epinephrine, norepinephrine, phenylephrine, vasopressin, dopamine, and dobutamine

WHC for Encephalopathy

Structured predefined entries for nursing assessments of speech, cognition, orientation, and level of consciousness documented in flowsheet data reports were extracted from the CDR using SQL queries. These entries are recorded by nursing staff as part of their admission, shift‐change, and per unit‐determined nursing assessments, as mandated by the Joint Commission in the standards for Provision of Care, Treatment, and Services^{( 25 )} and the American Nursing Association’s Standards of Professional Nursing.^{( 26} , ^{27 )} HE grades were mapped based on matching these entries with clinical descriptors used for grading HE in practice guidelines and in the HE Scoring Algorithm (Table 2).^{( 28} , ²⁹ , ³⁰ , ^{31 )} Glasgow Coma Scores (GCSs) were also mapped with WHC grades, with GCS 15 mapping to grade 0, GCS 12‐14 mapping to grade 2, GCS 4‐11 mapping to grade 3, and GCS 3 mapping to grade 4.^{( 30 )} If there were multiple data entries (e.g., entries for GCS, level of consciousness, and orientation) recorded at a given time or if the entries mapped to discrepant WHC grades, the maximum mapped WHC grade was used by default to maximize detection sensitivity. Structured data entries that did not fall under the above criteria, such as “Other (comment),” and unstructured data entries were excluded from mapping, thereby avoiding the need for natural language processing.

TABLE 2.

Mapping of Flowsheet Data on Four Domains of Mental Status and GCS to WHC for HE*

WHC(28‐30)	Flowsheet Documentation of Mentation				GCS
	Speech	Cognition	Orientation Level	Level of Consciousness
Grade 0. "No encephalopathy at all, no history of HE."	Clear Appropriate for developmental age Uses written communication Nods/gestures appropriately	Appropriate judgment Appropriate safety awareness Appropriate attention/concentration Appropriate for developmental age Follows commands No short‐term memory loss	Oriented ×4 (person, place, time, and situation) Oriented to place Oriented to time Oriented to person Oriented to situation Appropriate for developmental age	Alert Awake Responds to verbal	15
Grade 1. "Trivial lack of awareness, euphoria or anxiety, shortened attention span, impairment of addition or subtraction, altered sleep rhythm."	Delayed responses	Impulsive	Disoriented to situation
Grade 2. "Lethargy or apathy, disorientation for time, obvious personality change, inappropriate behavior, dyspraxia, asterixis."	Word finding difficulty Slurred	Poor safety awareness Poor judgement Poor attention/concentration Short‐term memory loss	Disoriented to person Disoriented to time	Lethargic	12‐14
Grade 3. "Somnolence to semi stupor, responsive to stimuli, confused, gross disorientation, bizarre behavior."	Incomprehensible Expressive aphasia Receptive aphasia	Unable to follow commands	Disoriented to place Disoriented ×4	Somnolent Responds to pain only Difficult to maintain arousal Confused	4‐11
Grade 4. "Coma."	Global aphasia			Obtunded Unresponsive	3

^*This classification scheme was based on guidance from descriptors in the WHC and criteria used in the HE Scoring Algorithm.^{( 28} , ²⁹ , ^{30 )}

To validate the mapped WHC grades, we conducted blinded physician manual chart reviews of the relevant sections (subjective findings, physical examination, and assessment and plan) of history and physical notes, progress notes, and discharge summaries of a random subset (20%) of patient admissions. The validators were blinded to the WHC HE grades determined by the flowsheet query to avoid contamination bias. Validation WHC grades were assigned based on descriptors in the subjective, physical examination, and/or assessment, and plan sections of the note were matched to clinical descriptions used in the American Association for the Study of Liver Diseases and EASL practice guidelines for management of HE.^{( 28 )} For example, physical examination findings of “lethargy” on chart review would be considered consistent with WHC grade 2, while findings of “arousable to voice only” or “grossly confused” would be consistent with WHC grade 3 (Supporting Table S1). If there were discrepancies between physical examination findings among different members of the provider team, then we used findings based on a hierarchical read based on the level of training. For example, the attending physician’s documented examination finding would be used over that of a resident physician. This blinded‐physician review was conducted at 3 time points during each admission: at time of initial admission, during the hospitalization (defined as maximum value acquired during the admission), and at the time of discharge.

Determination of Oxygen Fraction

Whenever available, the recorded oxygen fraction (FiO₂) in ventilator data or vital sign flowsheet reports were used. If such data were not available, then we estimated FiO₂ based on nasal cannula and high‐flow nasal cannula flow rates, assuming closed‐mouth breathing as validated in respiratory care literature (Supporting Table S2).^{( 32} , ³³ , ^{34 )} These recorded and estimated FiO₂ values were used to calculate oxygen saturation/FiO₂ and partial pressure to fraction of oxygen ratios, when appropriate.

ACLF Definitions and Prognostication Score Calculation

For each patient admission, we used the above data and mapped WHC grades to diagnose ACLF and calculated prognostication scores based on those published by the NACSELD‐ACLF^{( 10 )} and CLIF‐C‐ACLF.^{( 13 )} The NACSELD‐ACLF score (range 0‐1) predicts the probability of 30‐day survival in hospitalized patients (Supporting Table S3).^{( 10 )} Model for End‐Stage Liver Disease (MELD) and MELD‐Na scores were also calculated, as described,^{( 35} , ^{36 )} as inputs into the NACSELD‐ACLF score. Similarly, the CLIF‐C‐ACLF score is a composite score (range, 0‐100), with a score ≥70 predicting up to 100% mortality at 28 days (Supporting Table S3).^{( 13 )} We did not use the APASL ACLF diagnostic criteria due to bacterial infection being the most common precipitant of ACLF in patients with cirrhosis in the United States, in contrast to those in the Asia‐Pacific region.^{( 19 )}

Using the mapped WHC grades from the methods above, we then generated automated diagnoses based on NACSELD and EASL‐CLIF criteria at the 3 time points specified for validation of WHC grades of HE: at the time of initial admission, during the hospitalization (defined as maximum value acquired during the admission), and at the time of discharge. To validate the automated ACLF diagnoses, we conducted blinded‐physician chart reviews of the relevant sections (physical examination, laboratory findings, and assessment and plan) of history and physical notes, progress notes, and discharge summaries of the same random subset (20%) of patient admissions validated in the WHC grade validation. NACSELD and EASL‐CLIF ACLF diagnoses were confirmed based on descriptors in the physical examination findings, laboratory values, and diagnoses in the assessment and plan sections of the notes as matched to the relevant diagnostic criteria.^{( 10} , ^{13 )}

Statistical Analyses

Clinical characteristics and laboratory data for participants were summarized by medians and interquartile ranges (IQRs) for continuous variables or numbers and percentages for categorical variables. Comparisons among groups were performed using chi‐square and Kruskal‐Wallis tests, where appropriate. We calculated sensitivities and specificities of the ability of the mapped WHC grades to detect any HE (WHC grades 1‐4) and severe HE (WHC grades 3‐4) uncovered by chart review. To rate interobserver agreement between mapped WHC grades and those acquired from chart review, we calculated Cohen’s kappa coefficients and generated 95% confidence intervals (CIs) from bootstrapping with 1,000 replications.^{( 37 )} Similarly, we also calculated sensitivities and specificities of the automated NACSELD‐ACLF and EASL‐CLIF ACLF diagnoses versus manual chart diagnoses. Two‐sided P < 0.05 was considered statistically significant in all analyses. Analyses were performed using STATA statistical software, version 16.1 (StataCorp, College Station, TX).

Results

Of the 1,918 patients in the FrAILT Study at the University of California, San Francisco, 480 patients (25%) had 1,321 admissions during the 5‐year study period. Of these 1,321 admissions, 233 occurred after liver transplantation, 194 were admissions for the liver transplantation surgical procedure, 65 hospital stays were ≤24 hours, and 318 transplantations took place before enrollment in the FrAILT Study. Of the 239 remaining patients meeting the inclusion criteria, we isolated one admission per patient for 239 patient admissions (Fig. 1).

Baseline Characteristics

Baseline characteristics of the 239 patients are presented in Table 3. Of these patients, 37% were women, 46% were non‐Hispanic white, and median age at admission was 60 years (IQR, 53‐65 years). The most common etiologies of cirrhosis were chronic hepatitis C (31%), alcohol‐associated liver disease (24%), nonalcoholic fatty liver disease (21%), autoimmune/cholestatic diseases (10%), and chronic hepatitis B (5%). The median MELD score at admission was 21 (IQR, 15‐29), and the median MELD‐Na score at admission was 25 (IQR, 17‐32).

TABLE 3.

Baseline Clinical and Demographic Characteristics

Characteristic		FrAILT Cohort (n = 239)
Age in years at first admission (IQR)		60 (53‐65)
Female (%)		88 (37)
Race/ethnicity (%)
	White	111 (46)
	Black	9 (4)
	Hispanic	81 (34)
	Asian	23 (10)
	Native American	5 (2)
	Other	10 (4)
Etiology of liver disease (%)
	Hepatitis C	75 (31)
	Alcoholic	57 (24)
	Nonalcoholic fatty	49 (21)
	AIH/PBC/PSC	25 (10)
	Hepatitis B	13 (5)
	Other etiologies	20 (8)
HCC (%)		71 (30)
Comorbidities (%)
	Hypertension	112 (47)
	Diabetes	71 (30)
	Coronary artery disease	9 (4)
	Stroke	2 (1)
MELD at admission (IQR)		21 (15‐29)
MELD‐Na at admission (IQR)		25 (17‐32)
NACSELD‐ACLF at admission		17 (7)
	NACSELD‐OF 0	179 (76)
	NACSELD‐OF 1	40 (17)
	NACSELD‐OF 2	16 (7)
	NACSELD‐OF 3	1 (0.4)
	NACSELD‐OF 4	0 (0)
CLIF‐C‐ACLF at admission		89 (40)
	CLIF‐C‐ACLF class 0	134 (60)
	CLIF‐C‐ACLF class 1	39 (17)
	CLIF‐C‐ACLF class 2	34 (15)
	CLIF‐C‐ACLF class 3	16 (7)
Total length of stay (IQR)		5 (2‐9)

Abbreviations: AIH, autoimmune hepatitis; HCC, hepatocellular carcinoma; NACSELD‐OF, North American Consortium for the Study of End‐Stage Liver Disease–Organ Failures; PBC, primary biliary cirrhosis; PSC, primary sclerosing cholangitis.

Validation of WHC for HE

Given that validation of WHC grades using EHR had not previously been performed, we randomly selected 51 patient admissions (21%) to undergo manual chart review to validate the WHC grades mapped from flowsheet data. Sensitivities and specificities (along with 95% CIs) for the presence of any HE and severe HE by comparing mapped WHC grades versus chart review are presented in Table 4. The sensitivities for the presence of any HE ranged from 92%‐97%, while those for severe HE was 100% at the 3 time points queried (initial time of admission, during hospitalization, and time of discharge). Specificities for the presence of any HE ranged from 76%‐95%, while those for severe HE ranged from 78%‐98% at the 3 time points queried. Cohen’s kappa coefficients for agreement between different WHC grades were 0.55 (95% CI, 0.33‐0.74) at the time of admission, 0.64 (95% CI, 0.49‐0.79) at the time of maximum value in the middle of the admission, and 0.72 (95% CI, 0.51‐0.90) at the time of discharge.

TABLE 4.

Sensitivity and Specificity of Mapped WHC Versus Chart Review

	True Positive	False Negative	False Positive	True Negative	Sensitivity	95% CI	Specificity	95% CI
Any HE (WHC 1‐4) at admission	12	1	9	29	0.92	0.64‐1.00	0.76	0.60‐0.89
Any HE (WHC 1‐4) during hospitalization	31	1	3	16	0.97	0.84‐1.00	0.84	0.60‐0.97
Any HE (WHC 1‐4) at discharge	11	1	2	37	0.92	0.61‐1.00	0.95	0.83‐0.99
Severe HE (WHC 3‐4) at admission	4	0	6	41	1.00	0.40‐1.00	0.87	0.74‐0.95
Severe HE (WHC 3‐4) during hospitalization	15	0	8	28	1.00	0.78‐1.00	0.78	0.61‐0.90
Severe HE (WHC 3‐4) at discharge	5	0	1	45	1.00	0.48‐1.00	0.98	0.89‐1.00

ACLF Diagnoses

For the NACSELD‐ACLF diagnostic criteria for ACLF, 17 patients (7%) were diagnosed at the time of initial admission, with 16 having two organ failures and 1 having three organ failures. The number of patients diagnosed with ACLF by the NACSELD criteria increased to 64 (27%) during the hospitalization, with 20 having two, 18 having three, and 26 having four organ failures. At the time of discharge, 21 patients (9%) still had ACLF diagnoses, with 7 having two, 6 having three, and 8 having four organ failures. In comparison to the manual chart review of the same 51 patient admissions selected for WHC validation, we found the sensitivities and specificities for ACLF diagnoses under the NACSELD criteria to be 75% and 96%, respectively, at the time of initial admission. The sensitivity and specificities increased to 100% and 97%, respectively, during the admission and then to 100% for both at the time of discharge (Table 5).

TABLE 5.

Sensitivity and Specificity of Calculated NACSELD‐ACLF and CLIF‐C‐ACLF Diagnoses Versus Chart Review

Diagnosis	True Positive	False Negative	False Positive	True Negative	Sensitivity	95% CI	Specificity	95% CI
Dx of NACSELD‐ACLF at admission	3	1	2	45	0.75	0.19‐0.99	0.96	0.86‐1.00
Dx of NACSELD‐ACLF during hospitalization	15	0	1	36	1.00	0.78‐1.00	0.97	0.86‐1.00
Dx of NACSELD‐ACLF at discharge	4	0	0	47	1.00	0.40‐1.00	1.00	0.93‐1.00
Dx of CLIF‐C‐ACLF at admission	21	2	1	27	0.91	0.72‐0.99	0.96	0.82‐1.00
Dx of CLIF‐C‐ACLF during hospitalization	27	0	0	24	1.00	0.87‐1.00	1.00	0.86‐1.00
Dx of CLIF‐C‐ACLF at discharge	19	0	0	32	1.00	0.82‐1.00	1.00	0.89‐1.00

Abbreviation: Dx, diagnosis.

With respect to the EASL‐CLIF ACLF diagnostic criteria, 89 patients (40%) were diagnosed at the time of initial admission, with 39 meeting grade 1, 34 grade 2, and 16 grade 3 criteria. The number of patients diagnosed with ACLF by the EASL‐CLIF criteria increased to 114 (51%) during the hospitalization, with 28 meeting grade 1, 34 grade 2, and 52 grade 3 criteria. At the time of discharge, 76 (34%) still had ACLF diagnoses, with 35 meeting grade 1, 21 grade 2, and 19 grade 3 criteria. Similarly, when we compared these diagnoses to the manual chart review of the same 51 patient admissions selected for WHC validation, we found the sensitivities and specificities for ACLF diagnoses under the EASL‐CLIF criteria to be 91% and 96%, respectively, at the time of initial admission. These figures increased to 100% for both during the admission and at the time of discharge (Table 5).

Longitudinal ACLF Prognostication Scores

A total of 44,639 unique data points from the 239 patient admissions were available for ACLF prognostication score generation. This represented a median of 454 data points per admission (IQR, 194‐704) and a median of 28 data points per admission day (IQR, 21‐35). Using the data points generated from the relationally linked databases, we were able to calculate approximately hourly updated NACSELD‐ACLF and CLIF‐C‐ACLF scores. A representative example of one patient’s hospitalization course and the corresponding NACSELD‐ACLF and CLIF‐C‐ACLF scores are shown in Fig. 2.

FIG. 2.

Representative sample of calculated longitudinal NACSELD‐ACLF and CLIF‐C‐ACLF scores.

Discussion

In this study, we validated an informatics‐based method for capturing routine clinical care data from the EHR to calculate longitudinally updated ACLF prognostication scores in patients with ESLD hospitalized for ACLF. We extracted and mapped nursing assessment flowsheet data, which are captured at regular intervals, to WHC grades for HE and integrated these mapped values to more traditional data (laboratory values, device data, and medication administration records) to diagnose ACLF and calculate longitudinally updated prognostic scores under two definitions (NACSELD‐ACLF and CLIF‐C‐ACLF scores; Fig. 2). Our method demonstrated high sensitivity in detecting any or severe HE throughout the hospitalization and moderate sensitivity for diagnosing ACLF at admission that rapidly improved during the hospitalization.

While the techniques of using SQL queries of CDRs have been demonstrated,^{( 1 )} existing applications have been limited.^{( 7} , ⁸ , ^{9 )} Our method is novel in that it integrates the following multiple sources of EHR data: laboratory data, medication administration records, provider orders, ventilator device data, and flowsheet data. We made extensive use of flowsheets, which contain interprofessional (particularly nursing) assessments of mentation, functional status, and examination, that contained structured data entries that mapped to WHC grades based on previously validated instruments. Flowsheet data comprise one third of all recorded data in CDRs and have been historically underused in clinical research.^{( 3} , ^{4 )} Indeed, the linchpins in our methodology were mapping of flowsheet entries to appropriate WHC grades and of supplemental oxygenation flow rates to estimated FiO₂.

Validation of WHC grades generated from flowsheet data proved to have high sensitivity (92%‐95% for any HE and 100% for severe HE) versus clinicians’ documentation. Of note, HE grading has historically been difficult due to subjective assessments with poor to moderate interrater reliability.^{( 28} , ³⁸ , ³⁹ , ⁴⁰ , ^{41 )} This appears to be the rationale behind the use of overt HE, which has interrater reliability for diagnosing brain failure in both NACSELD and EASL‐CLIF definitions.^{( 38 )} Our calculations of Cohen’s kappa coefficient indicated moderate agreement between WHC grades mapped from flowsheet data and those rated by clinicians on retrospective chart review. While we found that Cohen’s kappa coefficients were lower at the beginning of the admission (0.55 vs. 0.72 at the time of discharge), this was thought to be due to lack of synchronization between clinician and nursing documentation, which gradually improved during the hospitalizations analyzed. The kappa coefficients from our study (0.55‐0.72) are within range of those reported for other methodologies for differentiating WHC grades.^{( 30} , ⁴² , ^{43 )} While the sensitivities and specificities of our methodology for ACLF diagnosis (based on NACSELD and EASL‐CLIF criteria) were imperfect at the time of initial admission, they rapidly improved to 100% as more information was generated and gathered throughout the admission.

We acknowledge the following limitations to our study. The first is that our methodology was developed at a single center and on a specific implementation of the EPIC EHR system without validation at another site. As such, our methodology and findings may not be readily generalizable to other centers (such as nontransplant centers) with non‐EPIC EHR systems. While specific structured data elements may differ in EHR flowsheets between institutions, the nursing documentation of neurologic and mentation assessments (level of consciousness, orientation, and cognition) used in our methodology are mandated nationally by the Joint Commission in the standards for Provision of Care, Treatment, and Services^{( 25 )} and the American Nursing Association’s Standards of Professional Nursing^{( 26} , ^{27 )} and locally by the California Nursing Practice Act.^{( 44 )} Review of charting and nursing standards of care documents at hospitals affiliated with our institution (academic medical center, county safety‐net hospital, and Veterans Affairs hospital) showed consistency in the detail of documentation in these assessments. Given prescribed standards for nursing assessments and documentations by national and local accreditation bodies, we anticipate that the data recorded in flowsheets are likely to be similar at other institutions and across EHR platforms. In our future works, we intend to replicate and validate this methodology at other institutions (using EPIC‐based data available from other University of California Health medical centers) and on other EHR platforms, such as Cerner.

Even with this first limitation, the major advantage of our methodology is that it uses existing documentation practices and charting infrastructure to discern different gradations of HE. While the exact execution for this methodology (e.g., SQL extraction code) will differ at another institution, the general strategy of mapping and extracting flowsheet data remains the same. In our experience, while SQL code written for our institution’s CDR does not often work out‐of‐the‐box against that of another institution, the differences are generally correctable and reconcilable. Moreover, ongoing efforts in reconciling flowsheet data to clinically focused information models, such as with regards to pain^{( 45 )} and genitourinary domains,^{( 46 )} using common data codes (e.g., Logical Observation Identifier, Name, and Codes and Systematized Nomenclature of Medicine)^{( 47} , ^{48 )} will allow for greater consistency across institutions. The movement toward standard data models, such as the Observational Medical Outcomes Partnership, which is used at more than 150 institutions and includes elements for flowsheet data, will also greatly improve future interoperability for the development of robust multicenter collaborations.^{( 49 )}

Second, the patient population evaluated in this study was highly selected because we only considered patients with ESLD who were enrolled in the FrAILT Study (evaluated for transplantation). This high degree of selection, however, was by design to validate our methodology in a controlled cohort. The next logical extension of this methodology is for implementations in larger and nonspecific cohorts, such as all patients with ICD‐9/10 discharge diagnoses of cirrhosis.

Third, we relied on blinded‐physician manual chart review of subjective findings, physical examination, and assessment and plans documented as the standard for validation. This is a limitation due to the retrospective nature of our study as many treating clinicians do not necessarily perform or document West Haven assessments in routine clinical care.

Last, given that the University of California, San Francisco Medical Center is a tertiary referral center, many of the admissions evaluated in our study were transferred to our medical center. Patient admissions in our sample, thus, may not reflect the initial clinical course. Moreover, we suspect that initial delays in documentation or clinician order placement (such as entering orders for dialysis) for transfer admissions contributed to the relatively poor sensitivity of our method for diagnosis of NACSELD ACLF at admission compared to chart review. As expected, the accuracy and precision of our methodology increased through the length of stay as more data were integrated.

Despite these limitations, this study serves as a proof of concept for a clinical informatics‐based methodology to generate longitudinally updated ACLF prognostication scores, which can better reflect the dynamic clinical course of these patients. Pilot demonstration of the validity of this methodology to extract accurate data in this population opens new analytic potentials, such as the application of big data methods that leverage the rich data from EHR platforms and CDR configurations to enhance investigation of predictors of outcomes in this dynamic population.

Supporting information

Table S1

Table S2

Table S3