Validating a novel algorithm to identify patients with autoimmune hepatitis in an administrative database

Abstract

Purpose:

Population-level studies on the treatment practices and comparative effectiveness of therapies in autoimmune hepatitis (AIH) are lacking due to the absence of validated methods to identify patients with AIH in large databases, such as administrative claims or electronic health records. This study ascertained the performance of International Classification of Diseases (ICD) codes for AIH, and developed and validated a novel algorithm that reliably identifies patients with AIH in health administrative data and claims.

Methods:

This was a cross-sectional study of patients with ≥1 inpatient or ≥2 outpatient ICD codes for AIH between 2008 and 2019 at a single health system. In a random sample of 250 patients, definite or probable AIH was determined using the Simplified AIH score, Revised AIH score or expert adjudication. The positive predictive value (PPV) was obtained. Variations of this base algorithm were evaluated using additional criteria to increase its performance.

Results:

Of the 250 patients, 143 (57.2%) patients had sufficient records available for review. The PPV of the base algorithm was 77.6% (95% CI: 69.9–84.2%). Exclusion of patients with ≥1 ICD code for primary biliary cholangitis or primary sclerosing cholangitis yielded a PPV of 89.7% (95% CI: 82.8–94.6%). Further exclusion of patients with recent immune checkpoint inhibitor therapy increased the PPV to 92.9% (95% CI: 86.5–96.9%).

Conclusions:

The use of ICD codes for AIH alone are insufficient to reliably identify patients with AIH in health administrative data and claims. Our proposed algorithm that includes additional diagnostic and medication-related coding criteria demonstrates excellent performance.

1 |. INTRODUCTION

Autoimmune hepatitis (AIH) is a rare inflammatory liver disease with a high risk of mortality.¹ While AIH is believed to affect all ages and races, there is significant heterogeneity in the occurrence of AIH worldwide.² This is not surprising given both genetic and environmental factors are hypothesized to play a role in the development of AIH. A majority of the available data on the epidemiology and treatment practices of AIH have arisen from studies conducted in Northern Europe and Asia. However, it has been suggested that the risk of disease progression and the barriers to care may be greater in racial and ethnic groups under-represented in European-based studies, such as Alaskan Natives, African Americans and Latino Americans.^3,4

The optimal management strategy of AIH has yet to be defined, and most current treatment recommendations are supported by a low level of evidence.⁵ Corticosteroids and azathioprine remain the standard of care for AIH since the 1970s, as the ability to conduct large clinical trials has been limited by the rare nature of this disease. Due to a clear relationship between therapy duration and the likelihood of sustained remission, AIH is frequently treated with long-term immunosuppression over many years. Unfortunately, nearly 80% of patients experience at least one treatment-related adverse event with a majority being due to the unintended effects of chronic corticosteroids.⁶

As part of the 21st Century Cures Act, the Food and Drug Administration formally recognizes the value of population-level observational research for healthcare decision-making.^7,8 However, there are no large-scale observational studies of AIH in the US outside of the inpatient setting.^9,10 A key reason for this has been the lack of a validated algorithm to reliably identify patients with AIH in readily available administrative databases. As a result, AIH treatment practices and outcomes in the United States remain largely unknown. The objective of this study was to develop and validate an algorithm that could serve as a reliable tool to facilitate future pharmacoepidemiology research efforts in AIH.

2 |. METHODS

This was a cross-sectional study of adults at University of Pennsylvania Health System (UPHS) inpatient and outpatient practices between October 1, 2008 and December 31, 2019. UPHS provides care at multiple sites in Philadelphia and the surrounding suburban areas, in rural and urbanized central Pennsylvania, as well as at multiple suburban sites in New Jersey. The health system includes tertiary care centers, one liver transplant center, and multiple community-level inpatient and outpatient facilities. From the electronic medical record (EMR), all patients with ≥1 inpatient or ≥2 outpatient International Classification of Diseases, Ninth or Tenth Edition, Clinical Modification (ICD-9-CM or ICD-10-CM) codes for AIH (571.42 and K75.4, respectively) were identified. There was no requirement for continuous health and/or drug plan enrollment required, or for a minimum duration of follow-up within UPHS to meet study inclusion. A random sample of 250 subjects were selected for manual chart review by a trained hepatologist (T.B.). To enhance the validity of the findings, a 10% sample was independently reviewed by a second hepatologist (N.M.). Both independent reviews led to similar adjudications (see section 3). The entirety of the patients' EMR was used for data extraction. All pertinent clinical, histopathologic, and laboratory data were evaluated, including scanned outside records. Demographic information was additionally collected including sex, race/ethnicity and age at possible AIH diagnosis (or at the time of first appearance of the ICD code for AIH if the former was not available). The presence of ≥1 inpatient or outpatient diagnostic code for primary biliary cholangitis (PBC)¹¹ and/or primary sclerosing cholangitis (PSC)¹² was also determined (Table 1). PBC and PSC are two autoimmune liver diseases involving the bile ducts that are distinct from AIH, but that can co-exist with AIH as an “overlap syndrome” in a small subset of patients.¹³ This is relevant as the accepted diagnostic criteria for AIH are not intended for use in patients suspected to have an overlap syndrome. Furthermore, the disease course and therapeutic management of such patients are inherently different.¹³

TABLE 1.

ICD-9-CM and ICD-10-CM codes for PBC and PSC

ICD-9-CM and ICD-10-CM for PBC	ICD-9-CM and ICD-10-CM for PSC
ICD-9-CM	ICD-9-CM
571.6	576.1
ICD-10-CM	ICD-10-CM
K74.3	K83.0
K74.4	K83.01
K74.5	K83.09
K75.5

Abbreviations: ICD-9/10-CM, International Classification of Diseases, Ninth/Tenth Edition, Clinical Modification; PBC, primary biliary cholangitis; PSC, primary sclerosing cholangitis.

The primary outcome was the presence of definite or probable AIH, as defined by the validated Simplified AIH diagnostic score,¹⁴ the Revised Original AIH score,¹⁵ or clinical adjudication. A flow diagram depicting the process used to ascertain the presence of the outcome of interest is shown in Figure 1. A simplified score ≥7 and of six were classified as definite and probable AIH, respectively.¹⁴ The modified histologic criteria by Balitzer et al¹⁶ were additionally used to score histopathologic features in the Simplified score calculation. If patients' Simplified AIH score was <6, the Revised Original AIH score was then calculated using the established thresholds: definite >17 and probable 12–17.¹⁵ Subjects with a Simplified score <6 and revised score <12 underwent expert adjudication (CL, DSG).

FIGURE 1.

Flow diagram depicting the methods used to identify patients with definite or probable autoimmune hepatitis

In reviewing patient charts, it was determined that a small subset of individuals had an immune-mediated liver injury that is associated with immune checkpoint inhibitor (ICI) therapy. These individuals had ICD codes for AIH present in their EMR with no evidence of true AIH disease.¹⁷ While medication data was not collected in the original identification of the cohort used for this study, it was hypothesized that the exclusion of patients receiving recent ICI treatment would further improve the PPV of the final algorithm. The National Drug Code (NDC) packaging codes for ICI therapies associated with this adverse effect are listed in Table 2, and those specifically used among the patients in our cohort are denoted.^17,18 The RxNorm Application Program Interface (API) browser from the National Library of Medicine was used to determine start date of NDC packaging codes and whether each code had been previously reused.^19,20

TABLE 2.

National drug codes for ICI therapies associated with an autoimmune drug-induced liver injury

Drug class	Nonproprietary drug name	NDC packaging code18	Start date of NDC code20
CTLA-4 inhibitor	Ipilimumab	0003-2327-11*	05/2011
		0003-2328-22*	05/2011
PD-1 inhibitor	Atezolizumab	50 242-917-01	06/2016
		50 242-917-86	12/2017
		50 242-918-01	04/2019
		50 242-918-86	04/2019
	Avelumab	44 087-3535-1	05/2017
	Durvalumab	0310-4500-12	06/2017
		0310-4611-50	06/2017
	Nivolumab	0003-3734-13	01/2018
		0003-3772-11	02/2015
		0003-3774-12	02/2015
	Pembrolizumab	0006-3026-02	03/2015
		006-3026-04	06/2019

Note: All NDC packaging codes listed active as of April 26, 2021. None of the NDC codes were previously reused during the 2008–2019 period.²⁰ Asterix denotes NDCs specifically identified in the EMR of patients with ICI therapy during manual chart review.

Abbreviation: ICI, immune checkpoint inhibitor; NDC, national drug code.

The demographic characteristics of the cohort were reported. These were also evaluated comparatively among those included for detailed review versus excluded due to insufficient records using chi-squared Kruskal–Wallis tests. The positive predictive values (PPV) and respective 95% binomial exact confidence intervals (CI) of three ICD-based coding algorithms were obtained. This study focused on PPV, as the objective was to develop a tool that would allow future investigators to accurately identify patients with AIH in an administrative database. Algorithm 1 required only ≥1 inpatient or ≥2 outpatient ICD codes for AIH. Algorithm 2 used this base algorithm and the absence of ICD codes for PBC and PSC, to allow for the exclusion of patients with a suspected overlap syndrome. Algorithm 3 built on Algorithm 2 and additionally excluded patients receiving an ICI ≤6 months prior to a diagnosis of AIH.¹⁷ As an exploratory analysis, the performance of Algorithm 3 according to sex and race/ethnicity was compared using descriptive statistics.

3 |. RESULTS

The overall demographics of the cohort were: 78.0% female, 60% White, 33.6% Black, 6.4% other race/ethnicity and with median age of 55 years (interquartile range: 42–64). Of the 250 random charts selected, 143 (57.2%) patients had sufficient records pertaining to their AIH diagnosis for review. The primary reasons for insufficient records were: (a) diagnosis preceded the start of the EMR and/or (b) gastroenterology care was obtained at an outside facility with no records available. There was no difference in sex, race/ethnicity or age between those with sufficient versus insufficient records (p = 0.5, p = 0.5, and p = 0.3, respectively).

Among these 143 patients, 36 (25.2%) and 28 (19.6%) had definite and probable AIH, respectively, according to the Simplified AIH criteria (Figure 2). Forty-seven (32.9%) had continued suspicion for AIH despite a Simplified score <6. Of these, eight met definite and 35 met probable criteria using the Revised AIH score.

FIGURE 2.

Flow diagram demonstrating the clinical outcomes identified in the random sample

Four subjects (2.8%) required expert adjudication and were deemed to have probable AIH. Clinical details are provided in Table 3. All four patients were followed in the EMR ≥2 years, had negative work-up for alternate diagnoses, and were managed as AIH by their treating physicians. Independent review of a 10% sample of charts by a second hepatologist (N.M.) demonstrated 100% agreement for the outcomes of definite AIH, probable AIH, and not AIH.

TABLE 3.

Clinical characteristics of four patients deemed as having probable AIH by expert adjudication

	Description	ALP:ALT ratio	ANA	ASMA	IgG	Biopsy	Clinical course
1	51yo F, no relevant history	<1.5	1:80	1:160	None	None performed	Treated with prednisone and AZA, relapse
2	46yo F, personal history of autoimmune disease	<1.5	Neg	Neg	Nl	“Mild chronic hepatitis” No additional detail available	Treated with prednisone and AZA with initial response, then relapse
3	42yo F, personal history of autoimmune disease	<1.5	Neg	Neg	Nl	“Minimal portal and lobular lymphocytosis” No additional detail available	Treated with prednisone and AZA with partial response
4	73yo M, no relevant history	<1.5	Neg	Neg	Nl	“Mixed hepatitis and cholestatic pattern of injury” No additional detail available	Treated with prednisone and AZA with initial response, then relapse

Abbreviations: ALP, alkaline phosphatase; ALT, alanine aminotransferase; ANA, anti-nuclear antibody; ASMA, anti-smooth muscle antibody; AZA, azathioprine; F, female; IgG, immunoglobulin G; M, male; Neg, negative; Nl, normal; yo, year old.

Nineteen (13.3%) had an AIH overlap syndrome. Of the 13 patients with no AIH suspected, autoimmune liver injury due to ICI therapy was the most frequent (N = 4), followed by sarcoidosis (N = 2), de novo AIH post-transplant (N = 2), nodular regenerative hyperplasia (N = 1), autoimmune cholangiopathy without evidence of overlap syndrome (N = 1) and unknown liver injury not consistent with AIH (N = 3). Of the four patients with immune-mediated liver injury due to ICI therapy, two had been treated with ipilimumab alone and two with combination therapy consisting of ipilimumab and nivolumab.

The PPV of Algorithm 1 for the identification of definite or probable AIH was 77.6% (95% CI: 69.9–84.2%; Table 4). After exclusion of patients with ICD codes for PBC and PSC, the PPV of Algorithm 2 was 89.7% (95% CI: 82.8–94.6%). Exclusion of patients with recent ICI treatment (Algorithm 3) further increased the PPV to 92.9% (95% CI: 86.5–96.9%). In exploratory analyses, there was no difference in the performance of Algorithm 3 by sex (95.5% female vs. 87.9% male; p = 0.3) or by race/ethnicity (92.2% White, 86.1% Black, 83.3% other; p = 0.5).

TABLE 4.

PPV of 3 proposed algorithms to identify probable and definite AIH

Algorithm Number	Presence of AIH ICD codea	Exclusion of PBC or PSC ICD codeb	Exclusion of PD1 or CTLA4 treatmentc	N	PPV for probable or definite AIH by Simplified score, Revised score or clinical adjudicationd, % (95% CI; N)
Algorithm 1	+	−	−	143	77.6 (69.9–84.2; 111)
Algorithm 2	+	+	−	117	89.7 (82.8–94.6; 105)
Algorithm 3	+	+	+	113	92.9 (86.5–96.9; 105)

Abbreviations: AIH, autoimmune hepatitis; ICD, International Classification of Diseases; PBC, primary biliary cholangitis; PPV, positive predictive value; PSC, primary sclerosing cholangitis.

^a≥1 inpatient or ≥2 outpatient instances of ICD-9-CM 571.42 and/or ICD-10-CM K75.4.

^b≥1 inpatient or outpatient instance of PBC diagnostic code (ICD-9-CM: 571.6; ICD-10-CM: K74.3, K74.4, K74.5, K75.5) or PSC diagnostic code (ICD-9-CM: 576.1; ICD-10-CM: K83.0, K83.01, K83.09).

^cEvidence of treatment with the following agents within 6 months of AIH diagnostic code: atezolizumab, avelumab, durvalumab, pembrolizumab, nivolumab, ipilimumab.

^dSimplified AIH Score ≥6 or Revised AIH Score ≥12.

4 |. DISCUSSION

In this study, we find that ICD codes for AIH alone are insufficient to reliably identify patients with true AIH using administrative data. We subsequently developed two novel algorithms that build on these ICD codes with very good performance. The first proposed algorithm requires the presence of ≥1 inpatient or ≥2 outpatient ICD codes for AIH in the absence of ≥1 inpatient or outpatient ICD code for PBC or PSC and was associated with a PPV of 89.7%. The second further excludes patients with evidence of ICI treatment, which can be achieved in outpatient pharmacy claims databases via the NDCs for these therapies shown in Table 2, leading to a PPV of 92.9%.

Despite the increasing availability over the last decade of large healthcare databases that can be leveraged for research, AIH remains an understudied disease, and particularly so in the United States. These results from three key issues: (a) nationwide population-level EMR databases are only available in select countries (e.g., Northern Europe); (b) the rare nature of AIH requires multicenter efforts to generate a sufficient sample size; and (c) the performance of the ICD codes for AIH was previously unknown. The findings from our study and our two novel algorithms will allow future investigators to more effectively study the “real world” management of patients with AIH, as well as the treatment-associated complications they face.

Our final algorithm iteration, Algorithm 3, used a combination of inclusion and exclusion criteria based on ICD codes, as well as exclusion of patients with NDC codes for ICIs, which are associated with an immune-mediated liver injury. While only patients treated with ipilimumab and nivolumab were observed in the conduct of our chart review, all five therapies listed have been associated with this particular type of liver injury¹⁷ and would be relevant to the identification of patients with AIH in healthcare claims databases. The NDC packaging codes listed can be used to identify ICI in outpatient infusion drug claims billed to health insurers (e.g., Medicare, Medicaid, private). Of note, these drugs are rarely, if ever, administered to hospitalized inpatients due to their high cost. According to the RxNorm API browser from the National Library of Medicine, all NDC packaging codes listed in Table 2 are active as of April 2021 and none have been previously reused for other drugs.^19,20 As newer ICI therapies become studied and marketed in the future, it would be important for investigators to adapt their algorithm to encompass these should there be a relevant risk of immune-mediated liver injury.

Implementation of this study was facilitated by the availability of two extensively studied and validated scoring systems to aid in the diagnosis of AIH for research and in clinical practice.^14,15 The Simplified score contains only four routinely measured parameters, while the Revised score is more complex with 15 different inputs. These diagnostic criteria include not only clinical and laboratory variables, but also histopathologic features as all patients with suspected AIH are expected to undergo liver biopsy.⁵ An advantage of our study, therefore, is the ability to confirm the presence of AIH from the EMR using these existing scoring systems. This reduced our dependence on expert adjudication, which may be subject to issues of both inter- and intra-rater reliability.

The reported PPVs were for definite or probable AIH. The PPV for definite AIH would be appreciably lower, but in clinical practice those patients with probable AIH are almost always treated as such. Indeed, all patients categorized as definite or probable AIH by the investigators of this study were being managed as de facto AIH by their gastroenterologist. Moreover, clinical trials and other research studies are frequently inclusive of either.²¹ These designations are felt to be reflective of differences in clinical presentation rather than in the accuracy of the diagnosis.²²

The validation of the proposed algorithms in a single health system is a limitation of this and most coding algorithm validation studies. The performance of our algorithms may have been impacted by the underlying case-mix of potential cases. The demographics of our AIH cohort was congruent with the previously published epidemiology of AIH in the United States, with the exception of having a slight over-representation of Black patients at UPHS.²³ It is unknown to what extent this impacts the generalizability of our algorithm with regards to research efforts within the US and internationally. Of note, we did not find any significant differences in the performance of Algorithm 3 according to sex or race/ethnicity.

UPHS provides both community-level and tertiary care services. While there is the possibility that more severe and/or complex patients were included in this cohort, the determination of probable or definite AIH is made at the time of diagnosis and is independent of patients' clinical course. Moreover, a substantial subset of the patients included were diagnosed with AIH at outside facilities, which may support our algorithms' external validity. This study was unable to determine whether patients met the exclusion criteria for the three algorithms if the diagnostic and/or medication billing coding only occurred at outside facilities and not at UPHS. For example, it is possible that some of the AIH patients may have had episodes of care that were associated with ICD codes for PBC or PSC that were missed. Nearly half of patients had insufficient records in their UPHS EMR to determine the outcome of probable or definite AIH according to the clinical criteria used in the Revised and Simplified scoring systems. However, the vast majority of these appeared to be receiving ongoing treatment for suspected AIH. The demographic characteristics of those excluded due to lack of records and those included in the chart review were also not different. Lack of detail in the reporting of histopathologic features on liver biopsy (67%) was a frequent reason for low to moderate Simplified AIH score.¹⁴ This is a known limitation of the Simplified AIH score, which requires specific terminology to meet the definition of “typical” on histopathology that can lead to underscoring of suspected AIH cases.¹⁶ This issue was ameliorated by the use of the modified histopathologic criteria, as well as the Revised AIH score, which provides more granularity with regards to histopathologic features observed.^15,16

In conclusion, our two proposed algorithms reliably identify patients with AIH in health administrative data. These will serve as important tools for future investigators seeking to advance the field of AIH research. In particular, these will allow for a more comprehensive assessment of the “real world” comparative effectiveness and safety of treatment options available to patients with AIH.

Key Points.

• The positive predictive value (PPV) of International Classification of Diseases (ICD) codes (≥1 inpatient or ≥2 outpatient) for autoimmune hepatitis (AIH) alone was 77%.

• The exclusion of patients with ICD codes for primary biliary cholangitis and/or primary sclerosing cholangitis increased the PPV to 90%.

• Further excluding patients with recent treatment with an immune checkpoint inhibitor led to a PPV of 93%.

• We therefore propose two novel algorithms that reliably identify patients with AIH in health administrative data.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.