Early Predictive Value of Different Indicators for Persistent Organ Failure in Acute Pancreatitis

Abstract

Goals:

In this study, we conducted this network meta-analysis (based on the ANOVA model) to evaluate the predictive efficacy of each early predictor.

Background:

Persistent organ failure (POF) is one of the determining factors in patients with acute pancreatitis (AP); however, the diagnosis of POF has a long-time lag (>48 h). It is of great clinical significance for the early noninvasive prediction of POF.

Study:

We conducted a comprehensive and systematic search in PubMed, Cochrane library, Embase, and Web of Science to identify relevant clinical trials, case-control studies, or cohort studies, extracted the early indicators of POF in studies, and summarized the predictive efficacy of each indicator through network meta-analysis. The diagnostic odds ratio (DOR) was used to rank the prediction efficiency of each indicator.

Results:

We identified 23 studies in this network meta-analysis, including 10,393 patients with AP, of which 2014 patients had POF. A total of 10 early prediction indicators were extracted. The mean and 95% CI lower limit of each predictive indicator were greater than 1.0. Albumin had the largest diagnostic odds ratio, followed by high-density lipoprotein-cholesterol (HDL-C), Ranson Score, beside index for severity in acute pancreatitis Score, acute physiology and chronic health evaluation II, C-reactive protein (CRP), Interleukin 6 (IL-6), Interleukin 8 (IL-8), Systemic Inflammatory Response Syndrome (SIRS) and blood urea nitrogen.

Conclusions:

Albumin, high-density lipoprotein-cholesterol, Ranson Score, and beside index for severity in acute pancreatitis Score are effective in the early prediction of POF in patients with AP, which can provide evidence for developing effective prediction systems. However, due to the limitations of the extraction method of predictive indicators in this study, some effective indicators may not be included in this meta-analysis.

INTRODUCTION

Acute pancreatitis (AP) is an inflammatory disease of the pancreas. It is the leading cause of hospitalization for gastrointestinal diseases worldwide, with a global total annual incidence of AP of 33.74 cases per 100,000 general population (95% CI: 23.33–48.81).^1,2 There was no statistically significant difference between males and females, and the disease mainly affected middle-aged or older adults.^3,4 AP is classified as mild, moderately severe, or severe depending on the extent of local damage in and around the pancreas and more importantly systemic damage to distal organs.⁵ Moderate and severe AP is often accompanied by local or systemic inflammatory complications, which are more prone to systemic organ dysfunction and later organ failure (OF).⁶ However, the incidence of OF varies widely among the reported patients with AP mainly due to differences in early diagnosis and early intervention.

AP patients with OF were classified by duration as persistent organ failure (POF) or transient organ failure. POF was defined as duration >48 h, while transient organ failure was ≤48 h.⁶ The cause of death in almost all patients with AP is OF. As the death caused by OF almost accounts for the mortality of all patients with AP, the mortality of transient OF is about 1.4%–10%, while the overall mortality OF POF is >40%.⁷ Patients with POF have a high risk of death in the first 2 weeks.⁶ However, it is a pity that the diagnosis of POF is time-delayed, which can be seen from its definition. Even if the patient is complicated with OF on day 1 of occurrence, it will take 48 hours to confirm whether the patient has POF or not. Early prediction of POF remains a clinical challenge. At present, many single indicators have certain predictive effects on POF, the most common of which are Acute Physiology and Chronic Health Evaluation (APACHE) II score, beside index for severity in acute pancreatitis (BISAP) score, C-reactive protein (CRP), Interleukin 6 (IL-6), Systemic Inflammatory Response Syndrome (SIRS), etc.⁷ . However, little work has been performed evaluating the predictive accuracy of these indicators.

Network meta-analysis was developed based on the traditional meta-analysis. Network meta-analysis is widely applied to evaluate a variety of intervention methods, and its striking feature is the use of rank probability and rank graphs to rank interventions.⁸ Recently, contrast-based model network meta-analysis has been gradually applied to evaluate the accuracy of diagnostic tests.^9,10 Nyaga¹¹ developed the diagnostic network meta-analysis based on the ANOVA model in 2018, which had advantages in the straightforward interpretation of the indicators. In this study, we adopt ANOVA model network meta-analysis to comprehensively summarize POF predictors and evaluate the predictors’ prediction efficiency.

MATERIALS AND METHODS

Literature Retrieval

A comprehensive search of published studies of pancreatitis complicated by POF was performed. We searched PubMed, Cochrane Library, Embase, and Web of Science from the inception of each database to September 29, 2021. The strategy of combining theme words (Mesh in PubMed) with keywords (Entry terms in PubMed) was adopted. PubMed search strategies were as follows: (organ failure [Title/Abstract]) AND ( ( (AP [Title/Abstract]) OR ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( (Pancreatitis [Title/Abstract])) OR (Pancreatitis, Acute Edematous [Title/Abstract])) OR (Acute Edematous Pancreatitides [Title/Abstract])) OR (Edematous Pancreatitides, Acute [Title/Abstract])) OR (Edematous Pancreatitis, Acute [Title/Abstract])) OR (Pancreatitides, Acute Edematous [Title/Abstract])) OR (Acute Edematous Pancreatitis [Title/Abstract])) OR (Pancreatic Parenchymal Edema [Title/Abstract])) OR (Edema, Pancreatic Parenchymal [Title/Abstract])) OR (Pancreatic Parenchymal Edemas [Title/Abstract])) OR (Parenchymal Edema, Pancreatic [Title/Abstract])) OR (Pancreatic Parenchyma with Edema [Title/Abstract])) OR (Pancreatitis, Acute [Title/Abstract])) OR (Acute Pancreatitis [Title/Abstract])) OR (Acute Pancreatitides [Title/Abstract])) OR (Pancreatitides, Acute [Title/Abstract])) OR (Peripancreatic Fat Necrosis [Title/Abstract])) OR (Fat Necrosis, Peripancreatic [Title/Abstract])) OR (Necrosis, Peripancreatic Fat [Title/Abstract])) OR (Peripancreatic Fat Necroses [Title/Abstract]))) OR (“Panc reatitis”[Mesh])).

Inclusion Criteria and Exclusion Criteria

Inclusion criteria included the following: (1) patients diagnosed with AP and POF were clearly defined; (2) sufficient information on the diagnostic value of 1 or more assessment indicators for POF; (3) reported in English; and (4) no restrictions on sex, age, or region.

Exclusion criteria included the following: (1) non-English literature; (2) Duplicate and irrelevant literature; (3) pure abstract paper; and (4) lack of true positive (TP), false positive (FP), false negative (FN), or true negative (TN).

Literature Screening Process and Information Extraction

In this study, 2 researchers (H.W., W.L.) independently conducted literature screening and data extraction and cross-checked after completion. If there is any dispute, a third researcher makes the judgment. The screening process mainly consists of 2 steps. First, the first round of screening is conducted according to the title and abstract to exclude reviews, in vitro trials, and studies with inconsistent topics. Second, download the literature conforming to the first round of screening, screen the included articles according to the full text, and determine whether to include the literature.

The extracted information mainly included study information (first author, publication year, country, etc.), patient information (number of patients, sex ratio, and mean age), evaluation indicators, a sample number of persistent organ failure, sensitivity, specificity, etc.

Selection of early prediction indicators included the following: (1) to avoid potential bias, we only further analyzed indicators reported in at least 2 studies; and (2) the indicators of TP, FP, FN and TN could be directly or indirectly extracted.

Quality Assessment

Two researchers (H.W. and W.L.) independently assessed the methodological quality of the included studies using quality assessment of diagnostic accuracy studies-2-2.¹² Disagreements in the quality assessments were resolved through discussion.

If necessary, a third independent researcher (M.L.) will join to make a final judgment. Quadas-2 consists of 4 crucial parts: summarize the review question, tailor the tool and produce review-specific guidance, construct a flow diagram for the primary study, and judge bias and applicability.

Statistical Analysis

Data Calculation

TP, FP, FN, and TN were calculated by a 2×2 contingency table of diagnostic tests if only sensitivity, specificity, the gold standard, and the number of POF cases were reported in the included study. If there are studies with multiple truncation values, TP, FP, FN, and TN with the best performance under truncation values are extracted. In addition, when only the ROC curve was included in the study, Origin software (2021 edition) was used to extract the sensitivity and specificity corresponding to the optimal threshold of the ROC curve and further calculate TP, FP, FN, and TN.

Network Meta-Analysis

To evaluate the accuracy of each indicator, the ANOVA model was applied for network meta-analysis.¹¹ The ANOVA model could also rank diagnostic tests by calculating the superiority index. Compared with the method based on the diagnostic odds ratio (DOR), the superiority index comprehensively considers the sensitivity and specificity, especially the diagnostic tests with high sensitivity and low specificity or low sensitivity and high specificity having advantages over DOR. We conducted a meta-analysis on R software (version 4.1.1, RSTAN (Package Version 2.21.3)). Based on the gold standard of POF diagnosis as the correlation mode of each predictor, the network graph of each predictor was drawn in Stata software (version 15.0).

RESULTS

Studies Selection Results

A total of 4630 related studies were retrieved, and 23^13–35studies were finally included for further analysis. The details of the literature screening are shown in Fig. 1.

FIGURE 1.

Flowchart of studies selection process.

Characteristics of Included Studies

The included studies were published between 2003 and 2021, and 16 were published between 2015 and 2021. The study included 10,393 patients with AP, of whom 2014 had POF. Fifteen of the included studies were retrospective, and the others were prospective. The studies included 10 predictors, namely APACHE II, BISAP, Ranson, high-density lipoprotein-cholesterol (HDL-C), Albumin (ALB), CRP, IL-6, IL-8, SIRS, and BUN (summarized in Table 1).

TABLE 1.

Characteristics of included studies.

					Sample size
No.	References	Years	Study design	Country	All	POF	Sex (Male/Female)	Age	Diagnostic criteria for AP	Indicator collection time	Predictors
1	Wu et al13	2021	retrospective	China	1848	142	1260/588	48.22 ±16.21	Atlanta criteria-2012	Within 48 hours after admission	3
2	Peng et al14	2020	prospective	China	309	43	196/113	50± 16	Atlanta criteria-2012	Within 24 hours after admission	1, 2
3	Chen et al15	2019	prospective	China	113	44	69/44	52.9 ± 1.4	Atlanta criteria-2012	Within 24 hours after admission	2
4	Zhou et al16	2018	retrospective	China	142	26	83/59	Range: 38–62.5	Atlanta criteria-2012	Within 24 hours after the onset of abdominal pain	4
5	Li et al17	2017	retrospective	China	158	46	85/73	48	Atlanta criteria-2012	Within 48 hours after admission	3, 4, 5
6	Cui et al18	2017	retrospective	China	105	21	62/43	Mean: 48.97	Atlanta criteria-2012	Within 6 hours after admission	3, 9
7	Chen et al19	2017	retrospective	China	208	47	121/97	47.5 ± 14.3	Atlanta criteria-2012	Within 24 hours after admission	2, 9
8	Mentula et al20	2003	retrospective	Finland	310	29	NA	NA	Atlanta criteria-1992	Unclear	1, 6
9	Chatzicostas et al21	2003	prospective	Greece	78	17	42/36	63.8 (25–93)	Atlanta criteria-1992	Within 48 hours after the onset of abdominal pain	1, 3
10	Wu et al22	2020	retrospective	China	1076	324	675/401	45	Atlanta criteria-2012	Unclear	3
11	Ueda et al23	2007	retrospective	Japan	54	30	41/13	52 ± 2	Unclear	Unclear	1, 3
12	Langmead et al24	2021	prospective	USA	133	37	63/70	50 (IQR:41)	Atlanta criteria-2012	Within 48 hours after the onset of abdominal pain	1–3, 6–10
13	Peng et al25	2015	prospective	China	66	35	34/32	59 (45–76)	Atlanta criteria-2012	Within 24 hours after the onset of abdominal pain	4
14	Qiu et al26	2019	retrospective	China	263	72	165/98	47.0(IQR:39–59)	Atlanta criteria-2012	Within 24 hours after admission	1
15	Kolber et al27	2018	prospective	Poland	95	12	65/30	48 ± 16.5	Atlanta criteria-2012	Within 24 hours after the onset of abdominal pain	7
16	Hong et al28	2017	retrospective	China	700	68	435/265	48 (37–63)	Atlanta criteria-2012	Within 24 hours after the onset of abdominal pain	5
17	Bertilsson et al29	2016	prospective	Sweden	92	13	51/42	Mean: 61	Atlanta criteria-2012	Within 14–24 hours after the onset of abdominal pain	1
18	Koutroumpakis et al30	2015	retrospective	USA	1612	294	822/790	53 (40–66)	Atlanta criteria-2012	Within 24 hours after admission	1, 10
19	Buddingh et al31	2014	retrospective	Dutch	115	97	71/44	60±16	Atlanta criteria-1992	Unclear	1, 6
20	Hong et al32	2013	retrospective	China	312	48	192/120	53 (42-65)	Atlanta criteria-2012	Within 36 hours after the onset of abdominal pain	1
21	Malmstrom et al33	2012	prospective	Denmark	60	45	26/34	60 (19–94)	United Kingdom guidelines for the management of acute pancreatitis-1998	Unclear	7, 8
22	Mofidi et al34	2007	retrospective	UK	664	77	NA	NA	Atlanta criteria-1992	Within 48 hours after admission	1
23	Li et al35	2020	prospective	China	1880	447	1123/757	Mean: 50.0	Atlanta criteria-1992	Within 6 hours after admission	6

The predictors column in Table 1 described the indicators reported in the included study, where 1–10 are APACHE II, BISAP, Ranson, HDL-C, ALB, CRP, IL-6, IL-8, SIRS, and BUN, respectively. ALB indicates albumin; BUN, blood urea nitrogen; CRP, C-reactive protein; HDL-C, high-density lipoprotein-cholesterol; IL-6, Interleukin 6; IL-8, Interleukin 8; SIRS, Systemic Inflammatory Response Syndrome.

Quality Evaluation Results

QUADAS-2 was used for method quality evaluation, and the evaluation results are shown in Fig. 2.

FIGURE 2.

A, Risk of bias graph; B, Risk of bias summary.

Meta-Analysis Results

Network Relationship Analysis

In this study, the network meta-analysis method was used to systematically evaluate the early prediction efficacy of different factors on POF. The gold standard of POF was used as the correlation mode of all predictors, and the Network diagram of predictors was drawn. Ten predictors were screened out, and the number of patients with AP and included studies reporting APACHE II score and Ranson score ranked first and second, respectively, as shown in Fig. 3.

FIGURE 3.

Network diagram of each predictive indicator and gold standard of POF diagnosis. ALB indicates albumin; BUN, blood urea nitrogen; CRP, C-reactive protein; HDL-C, high-density lipoprotein-cholesterol; IL-8, Interleukin 8; SIRS, Systemic Inflammatory Response Syndrome.

Sensitivity, Specificity, and DOR for Predicting POF

The meta-analysis results showed that the DOR of each indicator for POF was greater than 1, and 95% CI was inclusive of 1, which indicated that indicators included in this study had predictive efficacy for POF. Among them, ALB had the largest DOR [16.92 (95% CI: 4.59–26.99)], with a sensitivity of 70.17% (95% CI: 48.52–79.58) and specificity of 85.97% (95% CI: 74.65–91.68). The DOR of HDL-C, Ranson Score, BISAP score, and APACHE II were 11.13, 11.30, 10.80, and 10.17, respectively. (Table 2 and Fig. 4)

TABLE 2.

The sensitivity and specificity of each indicator were summarized.

		Sensitivity		Specificity		DOR
No.	Factors	Value	95% CI	Value	95% CI	Value	95% CI
1	APACHE II	80.10	74.13–83.77	71.25	66.93–75.73	10.17	6.88–13.87
2	BISAP	76.13	64.44–83.11	76.09	69.71–84.16	10.80	5.64–19.62
3	Ranson	81.45	74.29–86.24	70.93	65.46–78.75	11.13	7.07–18.21
4	HDL-c	63.19	54.96–82.27	83.21	77.08–92.39	11.30	5.12–34.28
5	ALB	70.17	48.52–79.58	85.97	74.65–91.68	16.92	4.59–26.99
6	CRP	68.40	50.96–74.44	80.64	73.88–87.52	9.74	4.17–14.61
7	IL-6	73.48	63.35–86.05	75.09	61.25–82.65	9.32	3.88–20.32
8	IL-8	63.57	53.53–82.83	74.14	63.95–88.03	6.81	2.83–23.57
9	SIRS	71.82	60.58–83.47	65.41	54.26–75.38	5.21	2.54–10.99
10	BUN	48.93	38.86–69.48	77.89	66.10–86.54	3.85	1.85–10.27

ALB indicates albumin; BUN, blood urea nitrogen; CRP, C-reactive protein; HDL-C, high-density lipoprotein-cholesterol; IL-6, Interleukin 6; IL-8, Interleukin 8; SIRS, Systemic Inflammatory Response Syndrome.

FIGURE 4.

The sensitivity and specificity of each indicator for early prediction of POF. ALB indicates albumin; BUN, blood urea nitrogen; CRP, C-reactive protein; HDL-C, high-density lipoprotein-cholesterol; IL-6, Interleukin 6; IL-8, Interleukin 8; SIRS, Systemic Inflammatory Response Syndrome.

Relative Sensitivity and Specificity

The diagnostic sensitivity and specificity of the APACHE II score were used as reference objects to analyze other indexes’ relative sensitivity and specificity. Analysis showed that compared with APACHE II, the sensitivity of other indicators did not increase significantly, while ALB, CRP, and IL-6 had better specificity. ALB had the most specificity advantage over the APACHE II score. The relative sensitivity and specificity are summarized in Table 3.

TABLE 3.

Relative sensitivity and specificity.

		Sensitivity		Specificity
No.	Factors	Value	95% CI	Value	95% CI
1	APACHE II	1	/	1	/
2	BISAP	0.73	0.49–0.88	1.05	0.91–1.22
3	Ranson	0.94	0.81–1.08	1.13	0.97–1.21
4	HDL-c	1.03	0.92–1.11	1.02	0.89–1.13
5	ALB	0.88	0.70–1.06	1.22	1.08–1.33
6	CRP	0.95	0.61–1.09	1.18	1.01–1.32
7	IL-6	0.86	0.64–0.95	1.11	1.02–1.24
8	IL-8	1.04	0.80–1.13	0.98	0.85–1.18
9	SIRS	0.92	0.70–1.06	1.07	0.90–1.26
10	BUN	0.96	0.77–1.07	0.89	0.74–1.06

ALB indicates albumin; BUN, blood urea nitrogen; CRP, C-reactive protein; HDL-C, high-density lipoprotein-cholesterol; IL-6, Interleukin 6; IL-8, Interleukin 8; SIRS, Systemic Inflammatory Response Syndrome

DISCUSSION

In this study, the ANOVA model was used to implement a network meta-analysis of the early prediction efficacy of POF; a total of 10 early prediction indicators of POF in patients with AP were summarized. The meta-analysis results showed that the predictors for early diagnosis of 10 kinds of POF and their 95% CI lower limit were all greater than 1, indicating that these predictors had certain early diagnosis values for POF. According to the results of DOR sequencing, ALB seemed to have the best early prediction performance. In addition, we used APACHE II as a reference. Relative analysis showed that compared with APACHE II, the sensitivity of other indicators did not increase significantly, while ALB, CRP, and IL-6 had better specificity. ALB had the most specificity advantage over the APACHE II score.

This study showed that the APACHE II, BISAP, Ranson, and SIRS scoring systems had relatively high sensitivity for the early prediction of POF, which is of great significance for predicting POF in patients with AP. However, due to the complexity of these scoring systems, it is not easy to use them widely in clinical practice.⁵ Il-6, IL-8, and CRP have been clinically evaluated to predict the outcome of acute pancreatitis.³⁶ This study showed that these indicators also had good accuracy in the early prediction of POF in patients with AP, which may be related to the severity of POF in AP. ALB seemed to have the highest DOR, but its sensitivity was lower than the other scoring systems. Recently, a systematic review by Yang et al³⁷ concluded that based on the diagnostic positive likelihood ratio, the Japanese Severity Scale and BISAP within 48 hours of admission and the Japanese Severity Scale and blood urea nitrogen within 48 hours of admission were the best predictors of POF. However, in our study, the prediction effect of blood urea nitrogen seemed not ideal, which may be due to the difference in the ranking method of prediction effectiveness. In summary, the early prediction of POF in patients with AP by all indicators included in this study seems to have only moderate accuracy, which indicates that although a single indicator prediction has good accuracy, it is more necessary to consider the development of a prediction system combining multiple indicators (such as a prediction model based on machine learning). Meanwhile, in addition to the early prediction indicators summarized in this study, there are other potentially effective prediction indicators.³⁸

This study had the following advantages: First, it was the first time to quantify the predictive efficacy of each index for the occurrence of POF in patients with AP by network meta-analysis, and the DOR reflected the best index for the early diagnosis of POF. Second, the total number of research samples included in this meta-analysis was relatively large. However, this study also had the following limitations: First, there was a lack of diagnostic randomized controlled trials in the studies included in this meta-analysis, which may bring some bias to the judgment of the results; Second: The methodology of this study did not consider the optimal truncation value of each indicator; Third, the selection of included indicators may not be comprehensive enough, because indicators reported in only 1 study and indicators that could not directly or indirectly extract TP, FP, FN, and TN were included; Fourth, network meta-analysis based on ANOVA model can only provide the results of sensitivity, specificity, DOR, relative sensitivity, relative specificity, and diagnostic advantage index. Further studies are necessary to calculate other diagnostic indicators.

CONCLUSION

The primary significance of this network meta-analysis is to summarize the early diagnostic indicators and efficacy of POF in patients with AP. Our findings show that ALB, HDL-C, Ranson Score, and BISAP Score are effective in the early prediction of POF in patients with AP, which can provide evidence for the development of effective POF early prediction systems (such as machine learning-based prediction models). However, due to the limitations of the extraction method of predictive indicators in this study, some effective indicators may not be included in this meta-analysis.