Clinical Prediction Models for Hepatitis B Virus-related Acute-on-chronic Liver Failure: A Technical Report

Abstract

Background and Aims

It is critical but challenging to predict the prognosis of hepatitis B virus-related acute-on-chronic liver failure (HBV-ACLF). This study systematically summarized and evaluated the quality and performance of available clinical prediction models (CPMs).

Methods

A keyword search of articles on HBV-ACLF CPMs published in PubMed from January 1995 to April 2020 was performed. Both the quality and performance of the CPMs were assessed.

Results

Fifty-two CPMs were identified, of which 31 were HBV-ACLF specific. The modeling data were mostly derived from retrospective (83.87%) and single-center (96.77%) cohorts, with sample sizes ranging from 46 to 1,202. Three-month mortality was the most common endpoint. The Asian Pacific Association for the Study of the Liver consensus (51.92%) and Chinese Medical Association liver failure guidelines (40.38%) were commonly used for HBV-ACLF diagnosis. Serum bilirubin (67.74%), the international normalized ratio (54.84%), and hepatic encephalopathy (51.61%) were the most frequent variables used in models. Model discrimination was commonly evaluated (88.46%), but model calibration was seldom performed. The model for end-stage liver disease score was the most widely used (84.62%); however, varying performance was reported among the studies.

Conclusions

Substantial limitations lie in the quality of HBV-ACLF-specific CPMs. Disease severity of study populations may impact model performance. The clinical utility of CPMs in predicting short-term prognosis of HBV-ACLF remains to be undefined.

Introduction

Acute-on-chronic liver failure (ACLF) is a clinically critical illness characterized by acute exacerbations of underlying chronic liver diseases with short-term high mortality.1,2 The etiology of underlying chronic liver diseases and precipitating events are distinct between Eastern and Western ACLF, which contributes to the heterogeneity of this syndrome.3 In Eastern ACLF, especially in China, hepatitis B virus related acute-on-chronic liver failure (HBV-ACLF) is the most common type.4

There are a variety of emerging therapies for HBV-ACLF, such as extracorporeal liver support device,5,6 glucocorticoid,7,8 granulocyte colony-stimulating factor (G-CSF),9 and cell therapies,10,11 but their efficacy requires further validation. Liver transplantation (LT) remains the only definite treatment to reduce the mortality of advanced HBV-ACLF12 but is limited by a lack of organ donors, huge financial cost of the procedure, and high mortality on the waiting list. In the European Association for the Study of the Liver–Chronic Liver Failure (EASL-CLIF) Acute-on-Chronic Liver Failure in Cirrhosis (CANONIC) study, ACLF patients had a 28-day mortality of 33.9%, and only 7.6% received LT.13 As a result, it is critical to precisely predict the short-term outcome of HBV-ACLF at the early stage of disease to make an accurate and prompt clinical decision of LT.

A number of clinical prediction models (CPMs) have been used to predict the short-term prognosis of HBV-ACLF utilizing laboratory and clinical variables that can be easily obtained in clinical practice. Some were specifically developed for HBV-ACLF, while others were originally developed for end-stage liver diseases [for instance, the model for end-stage liver disease (MELD) score,14 MELD-sodium (MELD-Na) score15 and Child-Turcotte-Pugh (CTP) score16], acute liver failure [King’s College Criteria (KCC)17], and other critical illness with organ failures [sequential organ failure assessment (SOFA)18]. Despite the number of available CPMs, there is no consensus on the use of optimal models to predict HBV-ACLF outcome. In addition, there are major concerns about the heterogeneity of study populations as well as model quality. Therefore, in the study, we systematically assessed both the performance and quality of available HBV-ACLF CPMs. We also analyzed the factors associated with heterogeneity and their predictive performance among different studies.

Methods

Study search and selection

A keyword search was carried out on articles related to HBV-ACLF published in PubMed from January 1995 to April 2020. The search strategy was developed as follows: (HBV OR hepatitis B) AND (severe flares of chronic hepatitis B OR chronic severe hepatitis B OR severe flare-up, chronic hepatitis B OR hepatic failure OR severe hepatitis B OR severe acute chronic hepatitis B (CHB) exacerbation OR hepatic decompensation OR severe acute exacerbation OR liver failure OR acute-on-chronic liver failure OR ACLF OR acute liver failure) AND (mortality OR prognosis OR outcome). Two reviewers (YX and LY) independently screened the searched articles based on the title, abstract, and full text sequentially. Disputes were resolved by negotiation between the two reviewers.

We included articles reporting the development of an HBV-ACLF-specific CPM or those assessing the predictive performance of previously established CPMs in non-HBV-ACLF-specific patients.

In addition, the included studies had clearly defined endpoints and reported the statistical modeling approaches if an HBV-ACLF-specific CPM was developed. For inclusion, the CPM had to contain at least two independent variables.

The exclusion criteria were as follows: (1) other types of publications, such as letters and reviews; (2) samples including patients younger than 18 years of age or pregnant women; (3) reports of biomarker-based prediction models; (4) reports of cost-benefit models; (5) experimental studies; or (6) decision-analysis studies.

Data extraction

We extracted the following information for each of the included articles: (1) year of publication; (2) study design; (3) study registration if reported; (4) diagnostic criteria for HBV-ACLF; (5) baseline characteristics of the study population; (6) sample size; (7) number of deaths or LT if reported; (8) variables included in the new CPMs; (9) statistical approaches for model development; and (10) model validation.

All information was independently extracted by the two reviewers, and disputes were resolved by negotiation between them.

Model assessment

Quality of HBV-ACLF-specific models

As shown in Supplementary Table 1, a scoring system was established by weighting study design, number of patients recruiting centers, sample size, adjustment of confounding factors, reporting of LT, and model validation. Studies with scores of 5–6 were considered high quality, 3–4 medium quality, and 1–2 low quality.

Performance of the CPMs

The performance of the CPMs was evaluated by discrimination and calibration.19 Discrimination referred to how well the model distinguished individuals at high risk of an event from those at low risk of an event.19 Calibration referred to the accuracy of absolute risk estimation.19 To measure model discrimination, we extracted the area under the receiver operating characteristic curve (AUROC) from each study. Quantitative pooled analysis of the discrimination performance of a specific model reported in several studies was performed by summary receiver operating characteristic (SROC) curves using Review Manager 5.3. To measure calibration, information on the Hosmer-Lemeshow test was extracted.

Ethics approval and consent to participate

The ethics committee of the First Affiliated Hospital of Zhejiang University reviewed and approved this study. Written consent from patients or their authorized representatives was waived.

Results

Characteristics of all CPMs

A total of 4,261 related studies were retrieved from PubMed based on the keyword search. According to the inclusion and exclusion criteria, 52 studies were selected after being screened by the title, abstract, and full text (Fig. 1). A total of 52 articles were extracted, of which 31 developed HBV-ACLF-specific CPMs and the other 21 assessed previously established CPMs. As shown in Figure 2, the number of publications is rapidly increasing each year. The studies were published in a number of academic journals (n=30), the most frequent being Chinese Journal of Hepatology [5 (9.62%)], followed by Medicine (Baltimore) [n=4 (7.69%)].

Fig. 1. Flow chart of study selection.

Fig. 2. Cumulative growth in relevant publications on PubMed by April 14, 2020.

The diagnosis of HBV-ACLF in these studies was made mainly based on the Asian Pacific Association for the Study of the Liver (APASL) consensus for ACLF (51.92%) or the Chinese Medical Association (CMA) liver failure guidelines (40.38%). Among all studies, the sample size ranged from 46 to 1,202 patients. Significant heterogeneity was observed in patient characteristics among the different studies, as shown by the sex proportion (male/female) (ranging from 2.96 to 12.19), incidence of cirrhosis (24–100%), incidence of hepatic encephalopathy (10–51%), incidence of ascites (36–91%), and mean MELD score (20.97–29.00). The type of precipitating event was reported in seven studies (13.5%), with flare-up of hepatitis B being the major event in each study. Mortality varied among the different studies, with 3-month mortality ranging from 26% to 87%.

Regarding reporting of LT, 18 studies did not mention LT (34.62%), 21 excluded patients receiving LT (40.38%), and 5 defined LT and death as a composite endpoint (9.62%). LT was regarded as the censored event in six studies (11.54%). Patients with LT were defined as survivors in one study (1.92%). In one study, patients who received LT within 3 months were considered dead and more than 3 months as surviving.

In 8 studies (15.4%), dynamic parameters were used for modeling. ΔMELD or ΔMELD-Na calculated as the difference between MELD or MELD-Na at two time points was most frequent. One parameter was constructed based on the daily levels of predictive variables for 7 days after diagnosis combined with baseline risk factors. In the other studies, only baseline parameters were used.

Characteristics of HBV-ACLF-specific CPMs

Thirty-one CPMs were established specifically for HBV-ACLF (Table 1).

Table 1. Patient characteristics of HBV-ACLF-specific CPMs.

References†	Model	ACLF diagnostic criteria	Sample	Death events	Endpoint time	Basic characteristics of the study population at admission
						Age in years	Sex, male/female	Cirrhosis, n/total	Ascites, n/total	HE, n/total	TB in mmol/L	INR	MELD score
[1]	Ke’s model	CMA	205	104	NA	NA	NA	NA	NA	NA	NA	NA	NA
[2]	Li’s model	CMA	409	215	NA	42±12	378/31	NA	NA	NA	NA	NA	NA
[3]	Sun’s model	CMA	204	118	90-day	46.8±13.2	170/34	110/204	NA	86/204	318.6±175.8	NA	26.0±9.0
[4]	LRM	APASL	452	175	90-day	45.6±11.5	361/91	138/452	334/452	119/452	NA	NA	NA
[5]	He’s model	CMA	172	75	90-day	45.16±11.21	144/28	132/172	96/172	NA	297.8±109.3	2.4±0.7	26.4±4.2
[6]	TPPM	APASL	248	133	90-day	42.27±11.98	225/23	68/248	152/248	95/248	270.9±140.3	2.0±0. 5	20.97±5.83
[7]	Zheng’s model	APASL	726	371	90-day	43.5±11.6	635/91	NA	530/726	251/726	NA	NA	NA
[8]	ALPH-Q	APASL	214	81	90-day	NA	160/54	99/214	123/214	45/214	NA	NA	NA
[9]	Yan’s model	APASL	432	209	90-day	46.9±13.3	329/103	239/432	348/432	115/432	351 (210)	2.8 (1.6)	27.8 (8.3)
[10]	Yi’s model	APASL	392	218	90-day	NA	323/69	NA	NA	165/392	NA	NA	NA
[11]	Li’s model	CMA	338	129	90-day	44.7±10.1	268/70	222/338	220/338	54/338	NA	NA	NA
[12]	HBV-ACLFs	EASL-ACLF	300	150	28-day	46.5±11.3	233/67	300/300	229/300	71/300	453.2±278.7	3.2±2.1	NA
[13]	HAM	APASL	530	190	90-day	41 (median)	489/41	246/530	264/530	95/530	NA	NA	NA
[14]	Chen’s model	APASL	551	241	90-day	NA	465/86	217/551	NA	NA	NA	NA	NA
[15]	MELD-LAC	AASL	236	106	90-day	NA	197/39	131 / 236	NA	NA	NA	NA	NA
[16]	HINAT ACLF	APASL	573	153 (28-day), 219 (90-day)	28-day, 90-day	43.5±11.5	478/98	NA	374/573	117/573	313.0±144.7	2.3±0.8	NA
[17]	Lei’s model	CMA	138	NA	the time of discharge or in-hospital death of the patient	45.80±11.01	111/27	51/138	96/138	NA	NA	NA	NA
[18]	Lin’s model	APASL	456	176	90-day	NA	383/73	NA	228/456	46/456	NA	NA	NA
[19]	Shi’s model	APASL	384	75 (30-day), 106 (60-day), 125 (90-day), 127 (180-day)	30-day, 60-day, 90-day, 180-day	NA	303/81	177/384	236/384	93/384	NA	NA	NA
[20]	Xue’s model	APASL	305	87	30-day	NA	257/48	89/305	212/305	92/305	NA	NA	NA
[21]	Gong’s model	CMA	184	75	90-day	NA	157/27	NA	122/184	NA	NA	NA	NA
[22]	Lin’s model	APASL	370	110	90-day	NA	314/56	88/370	248/370	103/370	NA	NA	NA
[23]	HINT	APASL	635	204	30-day	46.31±11.87	538/97	455/635	239/635	108/635	319.1 (220.9, 421.0)	2.02 (1.71, 2.55)	23.07±5.95
[24]	COSSH-ACLF	EASL-ACLF	657	233 (28-day), 313 (90-day)	28-day, 90-day	NA	586/71	466/657	366/657	130/657	NA	NA	NA
[25]	CTP-ABIC	CMA	222	80	90-day	NA	197/25	168/222	151/222	44/222	NA	NA	NA
[26]	Gao’s model	APASL	1,202	329 (28-day), 456 (90-day)	28-day, 90-day	NA	980/222	382/1,202	772/1,202	282/1,202	NA	NA	NA
[27]	APM	APASL	405	NA	28-day	NA	358/47	176/405	144/405	52/405	NA	NA	NA
[28]	ANN	APASL	402	160	90-day	47.2±13.3	316/86	NA	NA	NA	297.5±169.3	2.9±1.7	28.2±6.2
[29]	ANN	APASL	684	175 (28-day), 251 (90-day)	28-day, 90-day	43.9±11.6	582/102	NA	405/684	122/684	323.5±148.4	2.3±0.8	22.9 (20.0, 26.5)
[30]	CART	NA	777	316	90-day	NA	610/167	371/777	NA	NA	NA	NA	NA
[31]	CART	EASL-CLIF	489	191 (28-day)	28-day	NA	424/65	234/489	234/489	63/489	NA	NA	NA

^†See Supplementary File 1. LRM, logistic regression model; HBV-ACLF, hepatitis B virus related acute-on-chronic liver failure; MELD, model for end-stage liver disease; TPPM, Tongji prognostic predictor model; HAM, HBV-ACLF MELD; MELD-LAC, MELD-lactate; HINAT ACLF, HE-INR-NLR-age-TB ACLF; COSSH, Chinese Group on the Study of Severe Hepatitis B; CTP, Child-Turcotte-Pugh; ABIC, age-bilirubin-INR-creatinine; APM, artificial liver support system prognosis model; APLH-Q, age-prothrombin time-liver cirrhosis-hepatic encephalopathy-QTc; ANN, artificial neural network; CART, classification and regression tree; APASL, Asian Pacific Association for the Study of the Liver; CMA, Chinese Medical Association; AASL, American Association for the Study of Liver Failure; EASL-CLIF, European Association for the Study of the Liver–Chronic Liver Failure.

The diagnosis of HBV-ACLF in these studies was made mainly based on the APASL consensus [n=18 (58.06%)] or the CMA liver failure guidelines [n=8 (25.81%)]. EASL-ACLF criteria were used in four studies [n=2 (6.45%)] and Chinese Group on the Study of Severe Hepatitis B-acute-on-chronic liver failure (COSSH-ACLF) in one study [n=1 (3.23%)]. One study [n=1 (3.23%)] adopted the diagnostic criteria of acute liver failure proposed by the American Association for the Study of Liver Disease (AASLD). One study did not mention specific diagnostic criteria [n=1 (3.23%)].

As shown in Supplementary Table 1, 17 studies had a quality score of 0–2 (low quality), 12 had a score of 3–5 (medium quality), and only 2 had a score of 6–8 (high quality). Most were retrospective [n=26 (83.87%)] and single-center [n=30 (96.77%)], and only one was pre-registered. In terms of variable screening, most studies used regression approaches [n=26 (83.87%)]. The logistic regression model [n=14 (45.16%)] and the Cox hazard proportional model [n=12 (38.71%)] were the two methods most frequently used to identify risk variables. Two studies (6.45%) did not mention a clear variable screening method. Among the clinical variables consisting of CPMs, serum bilirubin (67.74%), international normalized ratio (INR) (54.84%), and hepatic encephalopathy (51.61%) were most frequent (Table 2). In terms of model formula, most CPMs were calculated as the results of multivariate logistic regression or Cox proportional hazard model as follows: (regression coefficients β1)×(variable 1)+(regression coefficients β2)×(variable 2)+(regression coefficients β3)×(variable 3)+….+constant (if logistic regression) (n=19 (61.29%). Three (9.68%) were calculated based on the sum of a series of categorical variables, the values of which were equally assigned [such as the Child-Turcotte-Pugh (CTP) score]; moreover, 5 (16.13%) were represented in the form of a nomogram, 2 (0.06%) were represented as an artificial neural network, and 2 (0.06%) were represented as a classification and regression tree.

Table 2. Variables consisting of model and screening approaches.

New CPMs	Variables	Methods
Ke’s model	TB; PTA; WBC; serum creatinine; maximum depth of ascites; HE score; singultus score; digestive tract hemorrhage score	Not mentioned
Li’s model	HE; serum creatinine; PTA; TB; infection; liver size; ascites fluid level	Clinical experience
Sun’s model	HR; LC; hepatitis B e antigen; ALB; PTA	Logistic regression
LRM	HE; HR; LC; hepatitis B e antigen; PTA; Age	Logistic regression
He’s model	HE; serum creatinine; INR; TB at the end of 2 weeks of treatment; cholinesterase	Logistic regression
TPPM	TB; INR; complications; HBV DNA	Logistic regression
Zheng’s model	TB; serum creatinine; PTA; HE; the maximum depth of ascites; WBC	Not mentioned
ALPH-Q	age; LC; PT; HE; QTc	COX regression
Yan’s model	age; HE score; MELD	COX regression
Yi’s model	HE; lnPTA2; lnINR2; lnTB2 (PTA2, INR2 and TB2 corresponded to those parameters at two weeks of treatment).	Logistic regression
Li’s model	age; Family history of HBV; HE; HR; WBC; PLT; INR; TB; TBA; CHE; serum creatinine; serum sodium; HBV DNA; hepatitis B e antigen	Logistic regression
HBV-ACLFs	age; serum creatinine; WBC	COX regression
HAM	MELD; HE; AFP; WBC; age	Logistic regression
Chen’s model	MELD, age, sodium	Logistic regression
MELD-LAC	LAC, MELD	Logistic regression
HINAT ACLF	HE, INR, NLR	COX regression
Lei’s model	NLR; serum levels of gamma-glutamyltransferase; ALB; sodium; artificial liver support therapy	Logistic regression
Lin’s model	age; LAAR; MELD	COX regression
Shi’s model	age; TB; serum sodium; PTA	COX regression
Xue’s model	TB; ALB; INR; Blood neutrophils percentage count; HE; Suspicion of infection	Logistic regression
Gong’s model	NLR; age; TB	COX regression
Lin’s model	TB; evolution of bilirubin; PTA; PLT; anti-HBe	Logistic regression
HINT	HE; INR; neutrophil count; TSH	COX regression
COSSH-ACLF	INR; HBV-SOFA; Age; TB	COX regression
CTP-ABIC	CTP; ABIC	COX regression
Gao’s model	age; TB; ALB; INR; HE	COX regression
APM	AFP; HE score; serum sodium; INR	COX regression
ANN	serum sodium; TB; age; PTA; Hb; hepatitis B e antigen	Univariate analysis and Artificial neural network
ANN	TB, PTA, serum sodium, HE, hepatitis B e antigen, GGT, ALP, age	Univariate analysis and Artificial neural network
CART	TB, age, serum sodium, INR	Univariate Logistic regression and Classification and regression tree
CART	HE, PT, TB	Logistic regression and Classification and regression tree

HE, hepatic encephalopathy; HB, hemoglobin; HR, hepatorenal syndrome; LC, liver cirrhosis; ALB, albumin; PTA, prothrombin activity; TB, total bilirubin; WBC, white blood cells; INR, international normalized ratio; PT, prothrombin time; QTc, the QT interval which is corrected for the heart rate; PLT, platelet; TBA, total bile acid; CHE, cholinesterase; AFP, alpha-fetoprotein; LAC, lactic acid; NLR, neutrophil–lymphocyte ratio; MELD, model for end-stage liver disease; LAAR, liver to abdominal area ratio; TSH, thyroid-stimulating hormone; GGT, γ-glutamyltransferase; ALP, alkaline phosphatase; LRM, logistic regression model; TPPM, Tongji prognostic predictor model; ANN, artificial neural network; HAM, HBV-ACLF MELD; MELD-LAC, model for end-stage liver disease-lactate; HINAT ACLF, HE-INR-NLR -age-TB ACLF; HINT, HE-INR-neutrophil count-thyroid stimulating hormone; COSSH, Chinese Group on the Study of Severe Hepatitis B; CTP, Child–Turcotte–Pugh; ABIC, age-bilirubin-INR-creatinine; CART, classification and regression tree; APM, artificial liver support system prognosis model; APLH-Q, age-prothrombin time-liver cirrhosis-hepatic encephalopathy-QTc; ANN, artificial neural network; CART, classification and regression tree.

A total of 19 CPMs (61.29%) were validated, including 1 model that was validated by two cohorts. Single-center and multicenter validation cohorts were used in 14 and 6 studies, respectively (a single-center cohort and a multicenter cohort were used for the CPM with two validation cohorts). Eight of fourteen single-center validation cohorts were derived from the same center as the modeling cohorts, and the other six cohorts were derived from external centers. The validation cohort was prospective in five studies (26.32%) and retrospective in fourteen studies (73.68%). The patients in the model cohort and validation cohort were recruited during the same period in two studies but not in the other sixteen studies; one study did not mention the timing of recruitment. The sample size of the validation cohort was generally smaller than the derivation cohort and ranged from 88 to 300 patients.

Characteristics of non-HBV-ACLF-specific CPMs

A total of 21 studies evaluated the performance of CPMs that were non-specific for HBV-ACLF. Eighteen were single-center studies (85.7%) and three were multicenter studies (14.3%). Ten models developed for other diseases were evaluated, including KCC for acute liver failure, age-bilirubin-INR-creatinine (ABIC) score for alcohol liver diseases, albumin-bilirubin (ALBI) score for liver cancer, CTP, modified Child-Turcotte-Pugh (mCTP) score, MELD, MELD-Na, updated MELD (UpMELD), and MELD excluding the international normalized ratio (MELD-XI) score for end-stage liver diseases.

Model performance

Among the 52 selected studies, 50 evaluated model predictive performance. Forty-six studies reported the AUROC, four studies reported the C-Index, and only five studies reported the Hosmer-Lemeshow test to assess model calibration.

Table 3 presents the discriminative performance of each CPM. The AUROC of all CPMs varied between 0.521 and 0.970, the sensitivity between 34% and 100%, and the specificity between 2.60% and 93.31%. The AUROC of 31 CPMs specific for HBV-ACLF ranged from 0.63 to 0.97, the sensitivity from 44.44% to 92.6%, and the specificity from 42.3% to 95.31%. As shown in Table 2, the MELD score was the most widely used CPM (44 studies), followed by the MELD-Na score (21 studies) and the CTP score (19 studies). The capacity of discrimination of MELD varied widely among different studies, as indicated by the AUROC (between 0.58 and 0.94), sensitivity (between 43.70% and 100%), specificity (between 63.8% and 90.2%), and optimal cut-off point (between 21 and 32 points). Likewise, a large variation in predictive performance was seen in the MELD-Na score [AUROC (between 0.563 and 0.922), sensitivity (between 41.90% and 86.4%), specificity (between 61.9% and 86.7%), and optimal cut-off point (between 22.35 and 34.28)] and the CTP score [AUROC (between 0.553 and 0.878), sensitivity (between 34% and 99.35%), specificity (between 39.71% and 84%), and optimal cut-off point (between 9 and 12.5 points)].

Table 3. Discriminative performance of CPMs.

Model	AUROC/C-Index	Sensitivity	Specificity	Cut-off	References†
MELD	0.58–0.94	43.70–100%	63.8–90.2%	21–32	[3–6,8–10,12,13,15–46,51],
Ke’s model	NA	NA	NA	NA	[1]
KCC	0.642–0.783	41–59%	2.6–87.7%	0–0.5	[32,36]
CTP	0.553–0.878	34–99.35%	39.71–84%	9–12.5	[4,8–10,16–18,20,23,24,29,32,36,42,45–48],
MELD-Na	0.563–0.922	41.9–86.4%	61.9–86.7%	22.35–34.28	[5,13,14,16–18,20,22,24–29,34,37,39,46,47,49,52]
Li’s model	0.953	97%	82%	9.5	[2]
Sun’s model	0.647–0.891	68.6–72.3%	52.1–52.5%	−2.554	[3,4,13]
Zhang’s model(LRM)	0.68–0.914	64–92.6%	42.3–95.1%	–0.3264–0.5176	[3,4,8,13,30,36,41]
MELD-Na	0.521–0.886	41.9–78.21%	50.5–90.16%	25.6–32	[10,12,13,14,28,36,42,49,50]
He’s model	0.85±0.03	NA	NA	NA	[5]
iMELD	0.540–0.864	54.7–89.58%	56.16–85%	34.705–52	[5,10,13,14,17,28,31,36,37,39,42]
MESO	0.571–0.905	38.7–80.77%	75.25–91.80%	1.986–21.61	[5,10,13,28,42]
TPPM	0.786–0.970	84.09–89.6%	61.54–94.7%	0.22	[6,25,38]
Zheng’s model	0.900–0.970	NA	NA	NA	[7]
UpMELD	0.687	44.7%	87.2%	5.5	[39]
MELD-XI	0.647	55.3%	71.8%	20.5	[39]
UKMELD	0.766	57.6%	81.6%	45.5	[39]
ALPH-Q	0.837–0.896	78–78.7%	85.1%	6.778	[8]
Yan’s model	0.853–0.867	72–76%	84.8–89.2%	4.66	[9]
SOFA	0.705–0.751	54.2–60%	80.4–84.7%	6.5	[9,16]
CLIF-SOFA	0.711–0.876	54.3–80.14%	64.56–91.1%	7–8.5	[9,16,23,44,50]
Yi’s model	0.930±0.016	NA	NA	NA	[10]
iMELD-C	0.776–0.862	69.23–89.58%	78.71–80.33%	49.306–52.157	[10]
LRM	0.93	86%	87.1%	3.16	[11]
HBV-ACLFs	0.704 (C-Index)	NA	NA	NA	[12]
CLIF-C ACLFs	0.632–0.873	61.86–93.65%	63.7–78.6%	36.78–43.76	[12,16,23–27,29,31,44,46]
HAM	0.868–0.894	84.9–91.5%	70.9–75%	−1.191	[13]
mCTP	0.74	91%	48.8%	14	[42]
ALBI	0.583–0.784	62.2–65.9%	67.2–81.4%	–1.119–0.95	[17,43,45]
ALBI+MELD	0.912	76.7%	90.9%	NA	[43]
Chen’s model	0.867	NA	NA	NA	[14]
MELD-LAC	0.859	91.5%	80.1%	−0.4741	[15]
HINAT ACLF	0.839–0.855	82%	74.5%	4.6	[16]
CLIF-C OF	0.656–0.906	53.9–92.6%	72.9–78.8%	8.5–10.5	[16,24,25,44,45,46,50]
Lei’s model	0.656	62.2%	64.1%	NA	[17]
Lin’s model	0.854–0.890	NA	NA	NA	[18]
Shi’s model	0.790–0.799 (C-Index)	NA	NA	NA	[19]
Xue’s model	0.813–0.848	44.44%	93.63%	NA	[20]
ABIC	0.695–0.829	54.4–73.8%	81.7%	9.16–9.44	[45,48]
Gong’s model	0.63–0.742	NA	NA	NA	[21]
Lin’s model	0.79–0.86	67.3%	91%	−0.73	[22]
HINT	0.889–0.917	74.60–79.43%	84.56–95.31%	−0.77	[23]
COSSH-ACLF	0.718–0.898	54.9–89.04%	55.56–91.78%	3.7–6.4	[23–27,31,50]
CLIF AD	0.775	NA	NA	NA	[46]
CTP-ABIC	0.927	90%	80.3%	9.08	[48]
AARC-ACLFs	0.790	NA	NA	NA	[25]
Gao’s model	0.58–0.80 (C-Index)	NA	NA	NA	[26]
APM	0.747–0.790	73.2%	71.5%	2.56	[27]
ANN	0.765–0.869	NA	NA	NA	[28]
ANN	0.754–0.913	NA	NA	NA	[29]
CART	0.896–0.905	69.7–85.2%	80.1–93.5%	NA	[30]
CART	0.820–0.824	88.2–88.6%	62.7–68.5%	NA	[31]

^†See Supplementary File 1. CTP, Child–Turcotte–Pugh; KCC, King’s College Criteria; MELD, model for end-stage liver disease; SOFA, sequential organ failure assessment; LRM, logistic regression model; TPPM, Tongji prognostic predictor model; MESO, model for end-stage liver disease score to serum sodium ratio index; iMELD, integrated MELD model; UpMELD, updated MELD; MELD-Na, model for end-stage liver disease-sodium; MELD-Na, model for end-stage liver disease sodium; MELD-XI, MELD excluding the international normalized ratio; UKMELD, United Kingdom MELD; CLIF-SOFA, chronic liver failure-sequential organ failure assessment; iMELD-C, iMELD plus complications; HBV-ACLFs, hepatitis B virus related acute-on-chronic liver failure score; CLIF-C ACLFs, chronic liver failure-consortium acute-on chronic liver failure score; HAM, HBV-ACLF MELD; mCTP, modified Child-Turcotte-Pugh; ALBI, Albumin-bilirubin; MELD-LAC, model for end-stage liver disease-lactate; HINAT ACLF, HE-INR-NLR -age-TB ACLF; CLIF-C OF, chronic liver failure-consortium organ failure; ABIC, age-bilirubin-INR-creatinine; HINT, HE-INR-neutrophil count-thyroid stimulating hormone; COSSH-ACLF, Chinese Group on the Study of Severe Hepatitis B-ACLF; CLIF AD, chronic liver failure-consortium acute decompensation; AARC-ACLFs, APASL ACLF research consortium-ACLF; LRM-Z, Z logistic regression model; APM, artificial liver support system -prognosis model; APLH-Q, age-prothrombin time-liver cirrhosis-hepatic encephalopathy-QTc; ANN, artificial neural network; CART, classification and regression tree.

In addition, we performed a pooled analysis of diagnostic accuracy of several common CPMs. As shown by the summary receiver operating characteristic (SROC) curves in Figure 3, the overall discriminative performance of the MELD score and chronic liver failure-sequential organ failure assessment (CLIF-SOFA) score seemed to be higher than those of the CTP score and MELD-Na score.

Fig. 3. Relationship between MELD score on admission and AUROC values. MELD, model for end-stage liver disease; AUROC, area under the receiver operating characteristic curve.

AUROC, area under the receiver operating characteristic curve; MELD, model for end-stage liver disease.

Impact of ACLF severity and diagnostic criteria on model performance

To further analyze the factors contributing to the large variation in the predictive performance of a specific model among different studies, we compared the accuracy of MELD in HBV-ACLF defined by different diagnostic criteria. In APASL-defined ACLF patients, the AUROC of the MELD score was between 0.580 and 0.940, the sensitivity was between 43.7% and 88.9%, the specificity was between 67.2% and 90.2%, and the best cut-off point was between 21.57 and 29.6 points. In CMA-defined ACLF patients, the AUROC was between 0.612 and 0.906, the sensitivity was between 51% and 100%, the specificity was between 70.2% and 91.4%, and the best cut-off point was between 21 and 32 points.

Next, we assessed the relationship between the mean MELD value of patients at admission and the AUROC value of the MELD score. As shown in Figure 4, we found that the lower the mean MELD value of HBV-ACLF patients at admission, the greater the AUROC value. This suggested a negative correlation between disease severity at admission and the discriminative capacity of the MELD score.

Fig. 4. SROC for MELD score, CTP score, MELD-Na score, iMELD score, LRM score and CLIF-SOFA score.

SROC, summary receiver operating characteristic curve; MELD, model for end-stage liver disease; CTP, Child-Turcotte-Pugh; MELD-Na, MELD-sodium; iMELD, integrated MELD; LRM, logistic regression model; CLIF-SOFA, chronic liver failure-sequential organ failure assessment.

Discussion

In this study, we systematically summarized the available clinical prediction models for HBV-ACLF and performed an extensive review of each study with regard to modeling data, modeling approach and model performance. Although the number of HBV-ACLF-specific CPMs has increased rapidly in the past 10 years, there are major concerns about the quality and reproducibility of most of them. Our analysis showed that the development of most HBV-ACLF-specific CPMs was flawed in the quality of modeling data. Most studies were retrospective in nature, recruited patients from a single center, and had limited sample sizes. The model proposed by the Chinese Group on the Study of Severe Hepatitis B (COSSH) consortium is the only CPM that was developed on the basis of national, multicenter, and prospective cohort data. Nevertheless, the COSSH HBV-ACLF model is not fully validated, as the validation cohort is single center and not from external study centers. Another frequent weakness is the absence of information on LT or inappropriate handling of LT data. Generally, LT is regarded as a competing event with death. However, a competing risk model in survival analysis has seldom been used. Few of the studies reported the indication of LT when adopting the use of a composite endpoint that combined death and LT. Either using an LT-free cohort or defining LT as a censored event may underestimate the mortality of the overall population and introduce bias in model development.

The MELD score is recognized as the mainstay for evaluating end-stage liver disease.20 It was originally developed to predict the short-term prognosis of cirrhotic patients undergoing transjugular intrahepatic portosystemic shunt (TIPS).14 The present analysis showed that MELD is the most commonly used CPM for predicting HBV-ACLF outcome. However, a large variation in the discriminative performance of MELD as indicated by AUROC, sensitivity and specificity was observed in different studies. This variation raises the concern that the heterogeneity of the study populations may impact model performance. The population heterogeneity may be due to the use of different diagnostic criteria in various studies (Table 4). The current analysis suggests that the use of MELD in APASL- and CMA-defined HBV-ACLF patients can obtain comparable discriminative performance because both diagnostic criteria identify ACLF patients characterized by high bilirubin and coagulopathy. On the other hand, our findings reveal a wide range of AUROC values for the MELD score despite using the same inclusion criteria for HBV-ACLF. Even when specific criteria are used, HBV-ACLF cases represent a heterogeneous population. Defining the population is confounded by the type of precipitating events (for instance, flare-up of hepatitis, use of hepatotoxic drugs, large alcohol consumption and so on) and the severity of underlying chronic liver diseases (non-cirrhotic chronic liver disease or compensated cirrhosis).3,21,22 Our findings showed that a lower MELD at admission has higher predictive power in HBV-ACLF, and the use of MELD in those with ultra-high MELD scores achieves high predictive performance as well.23 These findings suggest that the severity of HBV-ACLF is another important confounding factor of model performance and that preferential inclusion of patients at both ends of the severity spectrum would overestimate the predictive capacity of models. In addition, both 28-day and 90-day mortality were used as primary endpoints in different studies, thus contributing to varying degrees of predictive performance. Death events occurred frequently between 28 days and 90 days post-admission but were less frequent after 90 days in APASL-defined ACLF.24–26 The CANONIC study, which defined 28-day mortality as the primary endpoint, also reported much higher mortality at 90 days in patients with ACLF grade 1 or 2.13 Therefore, the use of 90-day mortality as the primary endpoint better fits the natural history of ACLF.

Table 4. Similarities and differences of ACLF diagnostic criteria.

	CMA	APASL	EASL-CLIF	NACSELD	COSSH
Definition	Severe liver damage caused by various insults on the basis of chronic liver disease, representing a clinical syndromes mainly manifesting as coagulopathy, jaundice, hepatic encephalopathy, ascites, etc.	Acute hepatic insult manifesting as jaundice and coagulopathy. Complicated within 4 weeks by ascites and/or encephalopathy in a patient with previously diagnosed or undiagnosed chronic liver disease associated with high mortality.	An acute deterioration of pre-existing chronic liver disease usually related to a precipitating event and associated with increased mortality at 3 months due to multisystem organ failure.	A syndrome characterized by acute deterioration in a patient of cirrhosis due to infection presenting with two or more extrahepatic organ failure.	A complicated syndrome with a high short-term mortality rate that develops in patients with HBV-related chronic liver disease regardless of the presence of cirrhosis and is characterized by acute deterioration of liver function and hepatic and/or extrahepatic organ failure.
Proposing time	2006 (updated on 2014)	2009 (updated on 2019)	2013	2014	2017
Chronic liver disease	compensated chronic liver disease	Non-cirrhotic chronic liver disease and previously compensated cirrhosis	Decompensated cirrhosis	Decompensated cirrhosis	Non-cirrhotic chronic liver disease and cirrhosis
Acute precipitating events	Acute hepatic insults	Acute hepatic insults	Any and frequently without identifiable events	Infection	Any and frequently without identifiable events
Etiology	All	All	All	All	HBV
Definition of liver failure	PTA ≤40% and serum bilirubin ≥10 mg/dL or daily rise ≥1 mg/dL	INR ≥1.5 and serum bilirubin ≥5 mg/dL	Serum bilirubin ≥12 mg/dL	None	Serum bilirubin ≥12 mg/dL

CMA, Chinese Medical Association; APASL, Asian Pacific Association for the study of the liver; EASL-CLIF, European Association for the Study of the Liver-Chronic Liver Failure consortium; NACSLED, North American Consortium for the Study of End-Stage Liver Disease; COSSH, Chinese Group on the Study of Severe Hepatitis B.

The present study identified common variables used in CPMs, in addition to the components of MELD. The presence of hepatic encephalopathy (HE) was frequently reported to be an independent variable associated with poor outcome.27 In addition, indicators of systemic inflammation, such as white blood cells (WBC) count, neutrophil percentage, and neutrophil-to-lymphocyte ratio (NLR), are common risk factors for short-term death.28 Other common variables included age, presence of ascites, serum sodium and hepatitis B e antigen presence. On the other hand, one of the MELD parameters, serum creatinine, was less frequently reported as an independent risk factor in HBV-ACLF. As a result, the overall predictive performance of MELD in HBV-ACLF is not satisfactory, and consistent with this finding, recent studies have shown limited capacity of MELD-Na in identifying ACLF patients at high risk of death on LT waiting lists.29–31 By contrast, a MELD-based scoring system that integrates HE and age outperforms the MELD score in predicting 90-day mortality of HBV-ACLF.32 In addition to the variables constituting the CPMs, model performance is determined by the weighting of specific variables. For example, although MELD does not include important criteria such as HE and ascites, the CTP with these parameters performed less well overall than the MELD score in which each variable is equally weighted.

In conclusion, a growing number of HBV-specific CPMs have been developed in recent years, but most are flawed in either the quality of the modeling data, the integrity of the modeling approach, or external validation. The MELD score is the most commonly used CPM, although it is non-HBV-specific. However, there is significant heterogeneity in the predictive performance of the MELD score among different studies due to the confounding effect of disease severity. Therefore, the clinical utility of CPMs in predicting the short-term prognosis of HBV-ACLF remains to be undefined. There is redundancy in the current HBV-ACLF CPMs, and there is an urgent need to establish high-quality prognostic models to better guide clinical practice. The development of future HBV-ACLF-specific CPMs should include the following elements to ensure the reliability of the model: (1) unified HBV-ACLF diagnostic criteria with a defined endpoint; (2) high-quality and unbiased modeling and validation data from prospective, large-sample, multicenter cohorts, as well as real-world validation; (3) selection of a couple of non-redundant and easily accessible variables for inclusion in the model via a well-adjusted process; (4) appropriate handling of events competing with death; (5) assessment of model discrimination and calibration; and (6) appropriate presentation of clinical utility.

Abbreviations

AARC-ACLFs

APASL ACLF research consortium-ACLF

AASL

American Association for the Study of Liver Failure

ABIC

age-bilirubin-INR-creatinine

ACLF

acute-on-chronic liver failure

AFP

alpha-fetoprotein

ALB

albumin

ALBI

albumin-bilirubin

ALP

alkaline phosphatase

ANN

artificial neural network

APASL

Asian Pacific Association for the Study of the Liver

APLH-Q

age-PT-LC-HE-QTc

APM

artificial liver support system prognosis model

AUROC

area under the receiver operating characteristic curve

CART

classification and regression tree

CHE

cholinesterase

CLIF

chronic liver failure

CLIF AD

chronic liver failure-consortium acute decompensation

CLIF-C ACLFs

chronic liver failure-consortium acute-on chronic liver failure score

CLIF-C OF

chronic liver failure-consortium organ failure

CLIF-SOFA

chronic liver failure-sequential organ failure assessment

CMA

Chinese Medical Association

COSSH

Chinese Group on the Study of Severe Hepatitis B

CPMs

clinical prediction model

CTP

Child-Turcotte-Pugh

EASL-CLIF

European Association for the Study of the Liver–Chronic Liver Failure

G-CSF

granulocyte colony-stimulating factor

GGT

γ-glutamyltransferase

HAM

HBV-ACLF MELD

HB

hemoglobin

HBV

hepatitis B virus

HBV-ACLF

hepatitis B virus related acute-on-chronic liver failure

HE

hepatic encephalopathy

HINAT ACLF

HE-INR-NLR -age-TB ACLF

HINT

HE-INR-neutrophil count-thyroid stimulating hormone

HR

hepatorenal syndrome

ICU

intensive care unit

iMELD

integrated MELD model

iMELD-C

iMELD plus complications

INR

international normalized ratio

KCC

King’s College Criteria

LAAR

liver to abdominal area ratio

LAC

lactic acid

LC

liver cirrhosis

LRM

logistic regression model

LRM-Z

Z logistic regression model

LT

liver transplantation

mCTP

modified Child-Turcotte-Pugh

MELD

model for end-stage liver disease

MELD-LAC

MELD-lactate

MELD-Na

MELD-sodium

MELD-XI

MELD excluding the international normalized ratio

MESO

model for end-stage liver disease score to serum sodium ratio index

NACSELD

North American Consortium for the Study of End-Stage Liver Disease

NLR

neutrophil–lymphocyte ratio

PLT

platelet

PT

prothrombin time

PTA

prothrombin activity

QTc

QT interval corrected for heart rate

SOFA

sequential organ failure assessment

SROC

summary receiver operating characteristic curve

TB

total bilirubin

TBA

total bile acid

TPPM

Tongji prognostic predictor model

TSH

thyroid-stimulating hormone

UpMELD

updated MELD

UKMELD

United Kingdom MELD

WBC

white blood cells

Data sharing statement

No additional data are available.