Validation Study of Algorithms to Identify Malignant Tumors and Serious Infections in a Japanese Administrative Healthcare Database

ABSTRACT

BACKGROUND

This retrospective observational study validated case-finding algorithms for malignant tumors and serious infections in a Japanese administrative healthcare database.

METHODS

Random samples of possible cases of each disease (January 2015–January 2018) from two hospitals participating in the Medical Data Vision Co., Ltd. (MDV) database were identified using combinations of ICD-10 diagnostic codes and other procedural/billing codes. For each disease, two physicians identified true cases among the random samples of possible cases by medical record review; a third physician made the final decision in cases where the two physicians disagreed. The accuracy of case-finding algorithms was assessed using positive predictive value (PPV) and sensitivity.

RESULTS

There were 2,940 possible cases of malignant tumor; 180 were randomly selected and 108 were identified as true cases after medical record review. One case-finding algorithm gave a high PPV (64.1%) without substantial loss in sensitivity (90.7%) and included ICD-10 codes for malignancy and photographing/imaging. There were 3,559 possible cases of serious infection; 200 were randomly selected and 167 were identified as true cases after medical record review. Two case-finding algorithms gave a high PPV (85.6%) with no loss in sensitivity (100%). Both case-finding algorithms included the relevant diagnostic code and immunological infection test/other related test and, of these, one also included pathological diagnosis within 1 month of hospitalization.

CONCLUSIONS

The case-finding algorithms in this study showed good PPV and sensitivity for identification of cases of malignant tumors and serious infections from an administrative healthcare database in Japan.

BACKGROUND

Postmarketing surveillance provides valuable and objective information on the use of new therapies in real-world conditions. Although numerous methods are available for conducting postmarketing surveillance, well-designed postmarketing studies that utilize information from large administrative healthcare databases have the potential to be more cost-effective and time-efficient than conventional postmarketing surveillance in Japan [1]. The use of healthcare databases for postmarketing surveillance was endorsed by the Japanese Pharmaceuticals and Medical Devices Agency (PMDA) in accordance with the Ministerial Ordinance on Good Post-marketing Study Practice, which was revised and implemented on April 1, 2018 [2]. Subsequently, the PMDA has put forward guidance on basic concepts for the validation of outcomes definitions used in postmarketing database surveillance studies in Japan [3], and these databases are increasingly used to evaluate the safety and effectiveness of approved drug treatments, treatment patterns and costs, disease outcomes, and resource utilization in routine medical practice in Japan [1, 4].

Because administrative healthcare databases are not designed for postmarketing surveillance or the assessment of drug safety, validation studies are needed to confirm the extent to which case-finding algorithms can correctly identify patient populations and outcomes of interest [4, 5]. Positive predictive value (PPV), which is particularly important for comparative database analyses, and sensitivity, which is also important for surveillance purposes, are commonly used to measure the validity of case-finding algorithms. External validation is typically conducted using medical records as a gold standard [5]. Estimating the true number of cases in an administrative healthcare database is complicated by the risk of misclassification bias, particularly if classification relies on a single item such as a diagnosis code [6, 7]. Ideally, all cases in a database should be evaluated to determine the sensitivity of any case-finding algorithm; however, evaluation of all cases in a large database using detailed medical records is impractical. A potential solution is to limit the number of cases that require medical record review by generating a broad case-finding definition based on a code or combination of codes related to the disease of interest (i.e., diagnosis, procedures, and billing) and to conduct a medical record review only for those cases applicable to the definition [3]. As described in the Japanese Society for Pharmacoepidemiology task force report [8], a broad case-finding definition is assumed to include all true cases (i.e., “possible cases”); cases that do not meet the definition are not considered to be true cases and are not included in the medical record review. Sensitivity calculated based on “possible cases” is considered an approximation of the sensitivity of case-finding algorithms (pseudosensitivity).

Although many validation studies have been conducted in the United States and Europe, a recent review of validation studies published through 2017 in the Asia-Pacific region found limited English language publications, only six of which were conducted in Japan [5]. The current study was designed following discussions with the PMDA to validate case-finding algorithms for malignant tumor and serious infection in a Japanese administrative healthcare database using “possible cases” and medical record review as the gold standard [8]. As malignant tumor and serious infection are important clinical outcomes related to multiple disease states, including autoimmune diseases, the case-finding algorithms identified in this study will be useful for future postmarketing studies that analyze these outcomes in the administrative healthcare database.

METHODS

STUDY OVERVIEW

This was a retrospective observational study that was designed to validate case-finding algorithms for the detection of malignant tumor and serious infection from a commercially available administrative healthcare database (Medical Data Vision Co., Ltd. [MDV] database; Tokyo, Japan) [9] in accordance with the basic concepts for validating outcome definitions as set out by the Japanese Pharmaceuticals and Medical Devices Agency [3]. Validation of case-finding algorithms was conducted as follows (Fig. 1): (i) the analysis populations from the MDV database included “possible cases” identified by study investigators using a combination of International Statistical Classification of Diseases and Related Health Problems 10th revision (ICD-10) diagnosis codes and other procedural and billing codes for malignant tumor or serious infection (i.e., infection requiring hospitalization); (ii) true cases among a random selection of the “possible cases” were identified by medical record review using non-anonymized data by designated physicians at the participating hospitals; and (iii) pseudosensitivity and PPV were calculated by the study investigators to validate case-finding algorithms for malignant tumor and serious infection in the MDV database. The designated physicians reviewed non-anonymized medical records to confirm true cases, and anonymized data were sent by the participating hospitals to the study investigators for analysis. None of the study investigators or MDV personnel had access to patient medical records.

Fig. 1 . Study design.

“Possible cases” were those that met the conditions for basic condition 2 as shown in Table 1 for malignant tumors and Table 2 for serious infections (i.e., infections requiring hospitalization). Case-finding algorithms are defined in Table 3 for malignant tumors and Table 5 for serious infections.

ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision, MDV: Medical Data Vision Co., Ltd, PPV: positive predictive value.

This study was conducted in accordance with the Declaration of Helsinki and, in accordance with the Japanese Ethical Guidelines for Medical and Health Research Involving Human Subjects, informed consent was not required because data were anonymized. The study protocol, which included the use of non-anonymized medical records/Diagnosis Procedure Combination (DPC) data by the participating hospitals, was reviewed and approved by a central ethics committee (MINS Clinical Trial Review Committee).

DATA SOURCE

The MDV database contains anonymized administrative claims and DPC data of hospitalizations and outpatient visits at hospitals that participate in the DPC system. The DPC is a site-based classification system for reimbursement of inpatient care, with payments made on a flat-rate per day basis. Rates are specified for more than 5,000 DPC classifications that are defined by the ICD-10 diagnosis codes, procedure and billing codes, and other elements [10]. The MDV database covers approximately 24% of acute care hospitals in Japan and includes information on DPC submission and reimbursement, patient demographics (e.g., age, sex, height, and weight), ICD-10 diagnosis and procedure codes, laboratory tests, examinations, surgeries, treatments, and prescribed drugs [11].

The target DPC hospitals included in this study were identified on the basis of the availability of data, the feasibility of including the required outcomes, and the number of patients. Of the two eligible hospitals, one was a specialist oncology hospital, one was a general hospital, one hospital was small (<200 beds), and the other was medium sized (200–499 beds).

ANALYSIS POPULATION

The target populations for this study were randomized samples of patients who met the definitions for “possible cases” of malignant tumor or serious infection (i.e., infection requiring hospitalization) (Fig. 1). Identification of “possible cases” of malignancy and serious infection was confirmed separately within each hospital’s internal electronic database using the case-finding definitions provided by MDV. “Possible cases” were defined as patients with at least one hospital visit record from January 2015 to December 2017 with a relevant ICD-10 diagnosis code(s) for malignant tumor or serious infection AND with any of the additional conditions under study, including relevant codes for procedures and claims related to malignant tumor (Table 1, basic condition 2) or serious infections (Table 2, basic condition 2). The ICD-10 diagnosis and additional conditions used to identify “possible cases” are summarized in Additional file 1, Supplementary Tables S1 and S2.

Table 1. Definitions of malignant tumor.

Condition	Description
Basic condition 1	Having the date of the initial visit to the hospital or hospitalization for a clinical event of malignant tumor AND a confirmed malignant tumor diagnosis flag assigneda, with the first clinical event during the data extraction period being the only event to be studied
Additional condition 1	[tumor marker, histological malignant tumor, and other related tests (category code D)] within 1 month before or after the date of the initial visitb
Additional condition 2	[treatment/management of specified diseases (category code B)] within 1 month before or after the date of the initial visitb
Additional condition 3	[pathological diagnosis (category code N)] within 1 month before or after the date of the initial visitb
Additional condition 4	[photographing/imaging, etc. (category code E)] within 1 month before or after the date of the initial visitb
Additional condition 5	[surgery (category code K)] within 3 months after the date of the initial visitb
Additional condition 6	[drug prescription (limited to drugs designated in the drug code list)] within 3 months after the date of the initial visitb
Additional condition 7	[radiotherapy (category code M)] within 6 months after the date of the initial visitb
Additional condition 8	[anti-malignant tumor infusion (medical practice code list category code G)] within 3 months after the date of the initial visitb
“Possible cases” (Basic condition 2)	Basic condition 1 AND additional conditions (1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8)

^a Having an ICD-10 code for malignant tumor designated in the disease code list.

^b The date of initial visit to the hospital or hospitalization, including the day of the visit or hospitalization.

ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision.

Table 2. Definitions of serious infection.

Condition	Description
Basic condition 1	Having the date of hospital admission for a clinical event of infection AND a confirmed infection diagnosis flag assigneda, with the first clinical event during the data extraction period being the only event to be studied
Additional condition 1	[immunological infection test and other related tests (category code D)] within 1 month before the date of hospital admissionb and during hospitalization
Additional condition 2	[pathological diagnosis (category code N)] within 1 month before the date of hospital admissionb and during hospitalization
Additional condition 3	[photographing/imaging, etc. (category code E)] within 1 month before the date of hospital admissionb and during hospitalization
Additional condition 4	[drug prescription (limited to drugs designated in the drug code list)] during hospitalizationb
Additional condition 5	[intravenous infusion/injection (category code G)] during hospitalizationb
Additional condition 6	[surgery (category code K), abscess puncture (category code J), etc.] within 1 month before the date of hospital admissionb and during hospitalization
“Possible cases” (Basic condition 2)	Basic condition 1 AND additional conditions (1 OR 2 OR 3 OR 4 OR 5 OR 6)

^a Having an ICD-10 code for infection designated in the disease code list.

^b Including the day of hospital admission.

ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision.

There is no consensus on the required width of the confidence intervals for PPV and sensitivity. Therefore, a planned sample size of ≥100 true cases for this study was based on two chart validation studies that conducted medical record reviews in 75 patients with potential venous thromboembolism [12] and 103 patients with potential acute myocardial infarction [13] using the Sentinel Distributed Database developed by the United States Food and Drug Administration and a report from the Japanese Society for Pharmacoepidemiology [11]. To meet this sample size, 150 patient cases would be required to meet basic condition 2 based on the assumption that 90% of these would agree with the medical record review and be considered true cases. Similarly, at least 100 patient cases would be required to meet basic condition 2 based on the assumption that 70% of these would agree with the medical record review and be considered true cases. Using a sample size of 150 cases, 80% would be expected to meet the case-finding algorithm definitions with 70% sensitivity and a PPV of 78.8%, with a CI of ±10% for both sensitivity and PPV.

IDENTIFICATION OF TRUE CASES BY MEDICAL RECORD REVIEW

Designated physicians at the hospitals reviewed the medical records of the random sample of “possible cases” to determine whether the cases identified and the medical records were in agreement and could be considered true cases. The criteria for a true case of malignancy were considered and checked in the following order: a histological diagnosis, treatment received was for malignancy (including surgery, chemotherapy, radiation therapy, or palliative care), and diagnosis by imaging tests (including endoscopy or laboratory findings). The criteria for a true case of serious infection (i.e., infection requiring hospitalization) were considered and checked in the following order: treatment received was for serious infection (including surgery and symptomatic therapies); microbiological examination or antigen-antibody tests suggestive of infection; imaging tests suggestive of infection; and laboratory findings, urinalysis, and physical examinations suggestive of infection.

Medical record review was performed independently by two physicians experienced in the field of malignant tumor and by two physicians experienced in the field of infection, all of whom were affiliated with the participating hospitals. If the results of the review differed between the two physicians for a particular case, a third physician completed an independent assessment and made the final decision. Patient data for medical record review were collected using an anonymized case report form. For malignant tumors, the assessment used information collected for 6 months before and up to 2 years after the date of the first treatment or hospitalization during the study period. For serious infections, the assessment used information collected for 4 weeks before and after the date of the first hospitalization for the clinical event of infection during the study period.

STATISTICAL ANALYSIS

PPV, pseudosensitivity, and the corresponding 95% confidence intervals were calculated using the Wald test for each case-finding algorithm (Fig. 2). Subgroup analyses of PPV and sensitivity were performed by sex, age category, and type of malignancy for malignant tumor and by sex and age category for serious infection case-finding algorithms that provided high PPV without a substantial loss in sensitivity.

Fig. 2 . Calculation of pseudosensitivity and PPV.

A true positive (a) was a case identified by a case-finding algorithm and by medical record review and a false positive (b) was a case identified by a case-finding algorithm but not by medical record review. A false negative (c) was a case identified by medical record review but not by a case-finding algorithm and a true negative (d) was a case that was not identified by medical record review or by a case-finding algorithm. PPV was calculated as a/(a + b), and pseudosensitivity was calculated as a/(a + c) and was termed “sensitivity”.

NPV: negative predictive value, PPV: positive predictive value.

RESULTS

STUDY POPULATION

A total of 63,516 patients had at least one hospital visit at either of the two study hospitals from January 2015 to April 2017 or to January 2018 that was recorded in the MDV database (Fig. 3). Of the 2,940 potential patient cases from each hospital that met the criteria for “possible cases” of malignant tumor (Table 1, basic condition 2), 180 were randomly selected; of these, 108 were identified as true cases of malignant tumor based on medical record review (Fig. 3). Of the 3,559 potential patient cases that met the criteria for “possible cases” of serious infections (Table 2, basic condition 2), 200 were randomly selected; of these, 167 were identified as true cases of serious infections based on medical record review (Fig. 3).

Fig. 3 . Case selection.

“Possible cases” were those that met the conditions for basic condition 2 as shown in Table 1 for malignant tumors and Table 2 for serious infections. Case-finding algorithms are defined in Table 3 for malignant tumors and Table 5 for serious infections.

ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision, MDV: Medical Data Vision Co., Ltd.

EVALUATION OF MALIGNANT TUMOR ALGORITHMS

Of the 18 case-finding algorithms that were evaluated, algorithms 11 to 15 and algorithm 4 gave higher PPVs with no substantial loss in sensitivity (Table 3). Algorithm 11—which included the relevant diagnostic code plus the following additional conditions within 1 month before or after the date of the initial visit: tumor marker, histological malignant tumor, other tests; treatment/management of specified diseases; pathological diagnosis; and photographing/imaging—had a PPV of 60.0% and the highest sensitivity (100%). Inclusion of additional conditions (algorithms 12–15) did not improve PPV further. Furthermore, addition of drug prescription (additional condition 6) to a case-finding algorithm did not appear to increase PPV. Algorithm 4, which included the relevant diagnostic code and photographing/imaging within 1 month before or after the date of the initial visit, gave a high PPV (64.1%) without a substantial loss in sensitivity (90.7%) and was assessed further in the subgroup analysis.

Table 3. Validity of case-finding algorithms for malignant tumor in the MDV administrative healthcare database^a.

Case-finding algorithmb	Case definitions	True cases	Cases meeting definition	True positive cases	Pseudosensitivity		PPV
		(a + c)c	(a + b)d	(a)e	a/(a + c)f	95% CI	a/(a + b)	95% CI
1	BC 1 AND AC 1	108	59	37	34.3 (37/108)	25.3–43.2	62.7 (37/59)	50.4–75.1
2	BC 1 AND AC 2	108	42	35	32.4 (35/108)	23.6–41.2	83.3 (35/42)	72.1–94.6
3	BC 1 AND AC 3	108	108	74	68.5 (74/108)	59.8–77.3	68.5 (74/108)	59.8–77.3
4	BC 1 AND AC 4	108	153	98	90.7 (98/108)	85.3–96.2	64.1 (98/153)	56.5–71.7
5	BC 1 AND AC 5	108	23	18	16.7 (18/108)	9.6–23.7	78.3 (18/23)	61.4–95.1
6	BC 1 AND AC 6	108	30	26	24.1 (26/108)	16.0–32.1	86.7 (26/30)	74.5–98.8
7	BC 1 AND AC 7	108	5	5	4.6 (5/108)	0.7–8.6	100 (5/5)	100–100
8	BC 1 AND AC 8	108	0	0	0.0 (0/108)	0.0–0.0	NC (0/0)	NC
9	BC 1 AND (AC 1 OR 2)	108	84	59	54.6 (59/108)	45.2–64.0	70.2 (59/84)	60.5–80.0
10	BC 1 AND (AC 1 OR 2 OR 3)	108	133	86	79.6 (86/108)	72.0–87.2	64.7 (86/133)	56.5–72.8
11	BC 1 AND (AC 1 OR 2 OR 3 OR 4)	108	180	108	100 (108/108)	100–100	60.0 (108/180)	52.8–67.2
12	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5)	108	180	108	100 (108/108)	100–100	60.0 (108/180)	52.8–67.2
13	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5 OR 6)	108	180	108	100 (108/108)	100–100	60.0 (108/180)	52.8–67.2
14	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7)	108	180	108	100 (108/108)	100–100	60.0 (108/180)	52.8–67.2
15	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5 OR 6 OR 7 OR 8)	108	180	108	100 (108/108)	100–100	60.0 (108/180)	52.8–67.2
16	BC 1 AND (AC 1 OR 2 OR 5 OR 6 OR 7 OR 8)	108	99	70	64.8 (70/108)	55.8–73.8	70.7 (70/99)	61.7–79.7
17	BC 1 AND (AC 5 OR 6 OR 7 OR 8)	108	44	37	34.3 (37/108)	25.3–43.2	84.1 (37/44)	73.3–94.9
18	BC 1 AND (AC 3 OR 5 OR 6 OR 7 OR 8)	108	113	77	71.3 (77/108)	62.8–79.8	68.1 (77/113)	59.6–76.7

Bold = case-finding algorithms identified as best fit based on a balance of PPV and pseudosensitivity.

^a A random sample (n = 180) or 2,940 potential cases were analyzed.

^b Malignant tumor basic and additional conditions are defined in Table 1.

^c Number of cases identified as true by medical record review (i.e., sum of true positive and false negative cases).

^d Number of cases meeting the criteria for each case-finding algorithm (i.e., sum of true positive and false positive cases).

^e Number of cases meeting the criteria for each case-finding algorithm and identified as true by medical record review.

^f Pseudosensitivity was calculated as the number of true positive cases for each case-finding algorithm divided by the sum of true positive and false negative cases. However, any true cases that did not meet the criteria for “possible cases” may have been excluded from the analysis.

AC: additional condition, BC: basic condition, CI: confidence interval, MDV: Medical Data Vision Co., Ltd., NC: not calculable, PPV: positive predictive values.

In general, the PPV and sensitivity of malignant tumor algorithm 4 were consistent across the sex, age, and malignancy-type subgroups (Table 4). Across the malignancy types, PPV ranged from 66.7% to 83.3% and sensitivity ranged from 83.3% to 100% (Table 4). In elderly age categories (65 years and older), PPV (65.0%–74.1%) and sensitivity (90.7%–95.2%) were equal to or greater than the values seen in the overall population. However, PPV and sensitivity were lowest (48.7% and 82.6%, respectively) in patients <65 years of age (Table 4).

Table 4. Pseudosensitivity and PPV of malignant tumor case-finding algorithm 4^a in patient subgroups.

Subgroups	Possible cases	True cases	Cases meeting definition	True positive cases	Pseudosensitivity		PPV
		(a + c)b	(a + b)c	(a)d	a/(a + c)e	95% CI	a/(a + b)	95% CI
Male	90	54	81	51	94.4 (51/54)	88.3–100	63.0 (51/81)	52.5–73.5
Female	90	54	72	47	87.0 (47/54)	78.1–96.0	65.3 (47/72)	54.3–76.3
Age category
<65 years	57	23	39	19	82.6 (19/23)	67.1–98.1	48.7 (19/39)	33.0–64.4
≥65 years	123	85	114	79	92.9 (79/85)	87.5–98.4	69.3 (79/114)	60.8–77.8
65–74 years	57	42	54	40	95.2 (40/42)	88.8–100	74.1 (40/54)	62.4–85.8
≥75 years	66	43	60	39	90.7 (39/43)	82.0–99.4	65.0 (39/60)	52.9–77.1
Malignancy type
Solid tumor	118	89	109	83	93.3 (83/89)	88.1–98.5	76.2 (83/109)	68.2–84.2
Hematological	8	6	7	5	83.3 (5/6)	53.5–100	71.4 (5/7)	38.0–100
Colorectal	28	23	26	21	91.3 (21/23)	79.8–100	80.8 (21/26)	65.6–95.9
Gastric	16	10	15	10	100 (10/10)	100–100	66.7 (10/15)	42.8–90.5
Pancreatic	7	6	6	5	83.3 (5/6)	53.5–100	83.3 (5/6)	53.5–100
Hepatic	3	2	3	2	100 (2/2)	100–100	66.7 (2/3)	13.3–100
Breast	14	12	12	10	83.3 (10/12)	62.3–100	83.3 (10/12)	62.3–100

^a Malignant tumor algorithm 4 was defined as basic condition 1 (having the date of the initial visit to the hospital or hospitalization for a clinical event of malignant tumor AND a confirmed malignant tumor diagnosis flag assigned [ICD-10 code for malignant tumor designated in the disease code list], with the first clinical event during the data extraction period being the only event to be studied) and additional condition 4 ([photographing/imaging, etc. – category code E] within 1 month before or after the date of the initial visit to the hospital or hospitalization, including the day of the visit or hospitalization).

^b Number of cases identified as true by medical record review (i.e., sum of true positive and false negative cases).

^c Number of cases meeting the criteria for each case-finding algorithm (i.e., sum of true positive and false positive cases).

^d Number of cases meeting the criteria for each case-finding algorithm and identified as true by medical record review.

^e Pseudosensitivity was calculated as the number of true positive cases for each case-finding algorithm divided by the sum of true positive and false negative cases. However, any true cases that did not meet the criteria for “possible cases” may have been excluded from the analysis.

CI: confidence interval, ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision, PPV: positive predictive values.

EVALUATION OF SERIOUS INFECTION ALGORITHMS

Of the 15 serious infection case-finding algorithms, algorithms 1 and 7 had the highest PPV (85.6%) and nine algorithms had a sensitivity of 100% (Table 5). Addition of drug prescription (additional condition 4) to a case-finding algorithm did not appear to increase PPV. Algorithm 1 included the relevant diagnostic code and immunological infection test or other related tests within 1 month before the date of hospital admission, including the day of hospital admission, and during hospitalization. Algorithm 7 included the relevant diagnostic code and either immunological infection test/other related tests or pathological diagnosis within 1 month before the date of hospital admission, including the day of hospital admission, and during hospitalization. Algorithms 3 and 8, which both included photographing/imaging (additional condition 3), were also well balanced. However, algorithm 3 had slightly lower sensitivity, and inclusion of additional conditions, including photographing/imaging (case-finding algorithms 8–14), led to slight reductions in PPV (Table 5). Therefore, for simplicity, subgroup analyses were only conducted on algorithms 1 and 7.

Table 5. Validity of case-finding algorithms for serious infection in the MDV administrative healthcare database^a.

Case-finding algorithmb	Case definitions	True cases	Cases meeting definition	True positive cases	Pseudosensitivity		PPV
		(a + c)c	(a + b)d	(a)e	a/(a + c)f	95% CI	a/(a + b)	95% CI
1	BC 1 AND AC 1	167	195	167	100 (167/167)	100–100	85.6 (167/195)	80.7–90.6
2	BC 1 AND AC 2	167	52	41	24.6 (41/167)	18.0–31.1	78.9 (41/52)	67.8–90.0
3	BC 1 AND AC 3	167	187	162	97.0 (162/167)	94.4–99.6	86.6 (162/187)	81.8–91.5
4	BC 1 AND AC 4	167	166	140	83.8 (140/167)	78.3–89.4	84.3 (140/166)	78.8–89.9
5	BC 1 AND AC 5	167	158	140	83.8 (140/167)	78.3–89.4	88.6 (140/158)	83.7–93.6
6	BC 1 AND AC 6	167	1	1	0.6 (1/167)	0.0–1.8	100 (1/1)	100–100
7	BC 1 AND (AC 1 OR 2)	167	195	167	100 (167/167)	100–100	85.6 (167/195)	80.7–90.6
8	BC 1 AND (AC 1 OR 2 OR 3)	167	196	167	100 (167/167)	100–100	85.2 (167/196)	80.2–90.2
9	BC 1 AND (AC 1 OR 2 OR 3 OR 4)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
10	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
11	BC 1 AND (AC 1 OR 2 OR 3 OR 4 OR 5 OR 6)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
12	BC 1 AND (AC 1 OR 4 OR 5)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
13	BC 1 AND (AC 1 OR 4 OR 5 OR 6)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
14	BC 1 AND (AC 1 OR 2 OR 4 OR 5 OR 6)	167	200	167	100 (167/167)	100–100	83.5 (167/200)	78.4–88.6
15	BC 1 AND (AC 4 OR 5 OR 6)	167	188	159	95.2 (159/167)	92.0–98.5	84.6 (159/188)	79.4–89.7

Bold = case-finding algorithms identified as best fit based on a balance of PPV and pseudosensitivity.

^a A random sample (n = 200) or 3,559 potential cases were analyzed.

^b Serious infection basic and additional conditions are defined in Table 2.

^c Number of cases identified as true by medical record review (i.e., sum of true positive and false negative cases).

^d Number of cases meeting the criteria for each case-finding algorithm (i.e., sum of true positive and false positive cases).

^e Number of cases meeting the criteria for each case-finding algorithm and identified as true by medical record review.

^f Pseudosensitivity was calculated as the number of true positive cases for each case-finding algorithm divided by the sum of true positive and false negative cases. However, any true cases that did not meet the criteria for “possible cases” may have been excluded from the analysis.

AC: additional condition, BC: basic condition, CI: confidence interval, MDV: Medical Data Vision Co., Ltd., PPV: positive predictive values.

For both case-finding algorithms 1 and 7, PPV ranged from 83.5% to 90.1% for the sex and elderly age categories but was lowest (77.2%) in the <65 years age category (Table 6). Sensitivity remained at 100% across all sex and age category subgroups for both case-finding algorithms (Table 6). As algorithm 1 includes fewer variables, this algorithm is likely to be more easily applied than algorithm 7.

Table 6. Pseudosensitivity and PPV of case-finding algorithms 1 and 7 in patient subgroups.

Subgroups	Possible cases	True cases	Cases meeting definition	True positive cases	Pseudosensitivity		PPV
		(a + c)c	(a + b)d	(a)e	a/(a + c)f	95% CI	a/(a + b)	95% CI
Algorithm 1a
Male	89	76	86	76	100 (76/76)	100–100	88.4 (76/86)	81.6–95.2
Female	111	91	109	91	100 (91/91)	100–100	83.5 (91/109)	76.5–90.5
Age category
<65 years	59	44	57	44	100 (44/44)	100–100	77.2 (44/57)	66.3–88.1
≥65 years	141	123	138	123	100 (123/123)	100–100	89.1 (123/138)	83.9–94.3
65–74 years	38	32	37	32	100 (32/32)	100–100	86.5 (32/37)	75.5–97.5
≥75 years	103	91	101	91	100 (91/91)	100–100	90.1 (91/101)	84.3–95.9
Algorithm 7b
Male	89	76	86	76	100 (76/76)	100–100	88.4 (76/86)	81.6–95.2
Female	111	91	109	91	100 (91/91)	100–100	83.5 (91/109)	76.5–90.5
Age category
<65 years	59	44	57	44	100 (44/44)	100–100	77.2 (44/57)	66.3–88.1
≥65 years	141	123	138	123	100 (123/123)	100–100	89.1 (123/138)	83.9–94.3
65–74 years	38	32	37	32	100 (32/32)	100–100	86.5 (32/37)	75.5–97.5
≥75 years	103	91	101	91	100 (91/91)	100–100	90.1 (91/101)	84.3–95.9

^a Defined as basic condition 1 (the date of hospital admission for a clinical event of infection AND a confirmed infection diagnosis flag assigned [ICD-10 code for infection designated in the disease code list], with the first clinical event during the data extraction period being the only event to be studied) and additional condition 1 ([immunological infection test and other related tests – category code D] within 1 month before the date of hospital admission, including the day of hospital admission and during hospitalization).

^b Defined as basic condition 1 (the date of hospital admission for a clinical event of infection AND a confirmed infection diagnosis flag assigned [ICD-10 code for infection designated in the disease code list], with the first clinical event during the data extraction period being the only event to be studied) and additional condition 1 ([immunological infection test and other related tests – category code D] within 1 month before the date of hospital admission, including the day of hospital admission) or additional condition 2 ([pathological diagnosis – category code N] within 1 month before the date of hospital admission, including the day of hospital admission and during hospitalization).

^c Number of cases identified as true by medical record review (i.e., sum of true positive and false negative cases).

^d Number of cases meeting the criteria for each case-finding algorithm (i.e., sum of true positive and false positive cases).

^e Number of cases meeting the criteria for each case-finding algorithm and identified as true by medical record review.

^f Pseudosensitivity was calculated as the number of true positive cases for each case-finding algorithm divided by the sum of true positive and false negative cases. However, any true cases that did not meet the criteria for “possible cases” may have been excluded from the analysis.

CI: confidence interval, ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision, PPV: positive predictive values.

DISCUSSION

This is the first study to evaluate optimal case-finding algorithms for malignancy and serious infection in a Japanese administrative healthcare database. The PPVs and sensitivities of the case-finding algorithms reported in this study can be applied to a range of purposes, and different algorithms can be selected depending on the setting and study objectives. Several algorithms were identified that gave high PPV (64.1% for malignant tumor and 85.6% for serious infection) without substantial losses in sensitivity (90.7% for malignant tumor and 100% for serious infection) and may be most applicable for postmarketing safety studies. In addition, subgroup analyses gave consistent findings for both outcomes, except for the expected reduction in PPV in patients <65 years of age. As PPV has been shown to decrease with decreasing disease prevalence [14], the lower PPV in non-elderly patients is likely to have arisen because of a lower prevalence of malignancies and serious infections in this age group. In this study, approximately one-quarter of true cases were in non-elderly patients. These findings extend those from previous validation studies in Japan [4, 10, 15–20] and will facilitate future pharmacoepidemiological and postmarketing database studies that may require identification of malignant tumors and serious infections in the Japanese population.

Prioritizing sensitivity is important when the aim is to identify all target cases in a population and to reduce the likelihood of false negatives, such as is required for surveillance studies [21]. Although prioritizing PPV increases the likelihood of identifying true cases and reduces the likelihood of false positives, some target cases may still be missed [21]. Because of the trade-off between PPV and sensitivity [21], the optimal case-finding algorithm for postmarketing surveillance is considered to be one that provides a balance between PPV and sensitivity. However, broader algorithms could be used for safety surveillance to maximize detection of possible safety signals. In contrast to malignancy, almost all case-finding algorithms for serious infection in this study provided high PPV and sensitivity, suggesting that detection of true cases of serious infection was less susceptible to misclassification than cases of malignancy in the administrative healthcare database. In addition, the high PPV and sensitivity for the serious infection algorithms in this study are consistent with a previous validation study of healthcare-associated infection, in which the PPV and sensitivity of infections identified using antibiotic utilization patterns from hospital administrative data compared with chart reviews from four Japanese hospitals were 75% and 93%, respectively [17]. For malignancy, a relevant diagnostic code and photographing/imaging or at least four additional conditions gave an optimal balance between sensitivity and PPV. In particular, photographing/imaging (additional condition 4) was critical for optimizing PPV and sensitivity. However, the PPV of most case-finding algorithms for malignant tumor was slightly lower than previously reported for other administrative databases in Japan [15, 16]. This may be explained in part by the emphasis on sensitivity in our study, but it could also be because of differences in the prevalence of these conditions in the different databases and because individual patients are not tracked in the MDV database if they switch hospitals. In this study, medical record review of malignancy cases was collected for 6 months before and up until 2 years after the first treatment or hospitalization. However, even with this collection period, some patients may have received their diagnosis for malignancy at a different hospital from the one in which they were treated, which may have contributed to an increased number of false positive cases and lower PPV. A slightly lower PPV may also be because the definition of “possible cases” in this study included a combination of diagnosis, procedures, billing, and other codes related to the disease of interest; any true cases that did not meet these conditions may have been excluded from the analysis.

A key strength of this study was the use of a method to analyze a sample of cases that were identified using a combination of ICD-10 diagnostic codes and other procedural and billing codes (i.e., “possible cases”), which helped overcome the significant resource challenges associated with conducting validation studies and which was adopted in agreement with the Japanese regulatory authority. Consistent with the gold standard for validation studies, medical records were used for identification of true cases, and the outcomes of each review required agreement by at least two independent physicians. However, several limitations should be considered when interpreting the findings from this study. First, as PPV is influenced by sensitivity, specificity, and disease prevalence [14], the PPVs obtained in this study may not be generalizable to other populations. Second, although the MDV database contains a large elderly population (35% ≥65 years of age), which is appropriate for the assessment of malignant tumors, patient data from smaller or more specialist centers are not represented, which may have introduced a degree of bias in the study population. Third, physicians’ awareness of the “possible cases” when reviewing non-anonymized medical records may have influenced their decisions when confirming true cases. Finally, although the hospitals that participated in this study varied in size and type, there were only two; therefore, careful consideration should be made when generalizing the findings more widely across the entire MDV database.

CONCLUSIONS

In conclusion, this study has developed case-finding algorithms for malignant tumor and serious infection that can be used to identify treatment outcomes in an administrative healthcare database. These findings support the usefulness of administrative healthcare databases for postmarketing studies and will contribute to effective utilization of these resources.

ABBREVIATIONS

AC: additional condition, BC: basic condition, CI: confidence interval, CNS: central nervous system, DPC: Diagnosis Procedure Combination, ICD-10: International Statistical Classification of Diseases and Related Health Problems 10th revision, MDV: Medical Data Vision Co., Ltd., NC: not calculable, NOC: not otherwise classified, PMDA: Pharmaceuticals and Medical Devices Agency, PPV: positive predictive value.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

In accordance with the Japanese Ethical Guidelines for Medical and Health Research Involving Human Subjects, all data included in the analyses were anonymized and informed consent was not required. The study protocol was approved by a not-for-profit central ethics committee (MINS Clinical Trial Review Committee) on June 19, 2018.

CONSENT FOR PUBLICATION

This manuscript does not contain any images, videos, or personal data that require consent from individuals.

COMPETING INTERESTS

DK has no conflicts of interest to declare.

MS and MN are employees of Medical Data Vision Co., Ltd., Tokyo, Japan.

EY, AN, NT, KK, and TI are employees of Eli Lilly Japan K.K. and AN and TI are minor shareholders in Eli Lilly and Company.

FUNDING

This study was sponsored by Eli Lilly Japan K.K. and Medical Data Vision Co., Ltd. Eli Lilly Japan and Medical Data Vision were involved in the study design and in the collection, analysis, and interpretation of the data.

AUTHORS’ CONTRIBUTIONS

All authors contributed to the study design and interpretation of the study results, and participated in the drafting, critical revision, and approval of the final version of the manuscript. MS and MN participated in data collection, KK was involved in management of the study, and MS, MN, EY, and NT contributed to the statistical analyses.

AVAILABILITY OF DATA AND MATERIALS

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Supplementary Material

SUPPLEMENTARY INFORMATION