Evaluation of Commercially Available Helicobacter pylori Serology Kits: a Review

Several methods can be used to diagnose Helicobacter pylori infection. Most of them require upper gastrointestinal endoscopy for retrieval of a gastric biopsy specimen. For serology, no upper gastrointestinal endoscopy is required, but blood must be obtained to detect H. pylori antibodies. H. pylori serology is attractive in comparison with other diagnostic methods because it is simple, inexpensive, and less of a burden for the patient. Several kits for the detection of H. pylori by serology have become commercially available since the discovery of H. pylori by Warren (87) in 1983. Most of these H. pylori serology kits are based on various antibody preparations and different techniques.

The introduction of commercially available H. pylori kits has led to an increase in the number of studies that have evaluated kit characteristics. Recently, a systematic review comparing the accuracies of commonly used commercial serology kits for the detection of H. pylori infection has been conducted (48). To account for the different reference standards and designs used by various investigators, only studies that evaluated pairs of serology kits and that compared the kits only within those studies were included. A more appropriate method of comparing different diagnostic tests and the performance of different interpreters of one test is to calculate the area under the receiver operating characteristics curve (AURC) for each test (83). To correct for dependence between AURCs within the same study population, we used a random-effect model. By reviewing the literature, we also tried to determine whether H. pylori serology can accurately diagnose H. pylori infection. However, in contrast to the study by Loy et al. (48), we reviewed all the studies that evaluated commercially available H. pylori serology kits.

DATA COLLECTION AND EXTRACTION

Identification and eligibility of publications.

A computerized and manual literature search was performed in early 1998. Relevant publications were identified in MEDLINE (1983 to 1997) with the medical subject heading terms Helicobacter or pylori, Sero*, Sera*, Seru*, Sensitivity, and Human in checktags. Furthermore, additional publications were retrieved by reviewing references in publications found by MEDLINE. The criteria used to select publications were as follows: H. pylori infection was established before treatment; the H. pylori serology kit was commercially available; the number of patients, prevalence of infection, and the sensitivity and specificity of the H. pylori serology kit were described or could be calculated; and the studies were published in Dutch, English, French, or German.

Data analysis.

New diagnostic tests are mainly evaluated by determining the sensitivity and specificity of the test. For evaluative purposes, the sensitivity and specificity are less useful (83). On the basis of the receiver operating characteristic (ROC) curve, we calculated the AURC, which is a measure for the diagnostic performance of a test (42). It is independent of cutoff points and reasonably immune to selection bias. Depending on the serology kit result, one should use different methods to calculate the AURC. We used a method to estimate the AURC for kits with a quantitative test result by using one combination of a true-positive rate and a false-positive rate on the basis of the assumption that the data for the H. pylori-infected and noninfected persons were logistically distributed and had equal variances (84). On the other hand, for serology kits with a qualitative test result, we used the trapezium method to calculate the AURC (36). However, comparison of the H. pylori serology kits revealed that the diagnostic performance differed substantially depending on how the AURC was calculated. Therefore, we decided to estimate the AURCs by the trapezium method, irrespective of the distribution of the test result. Use of the trapezium method to estimate the AURC of a serology kit with a quantitative test result possibly underestimates its diagnostic performance (36).

The AURC was used to explore possible differences between clinical features of study populations and methodological aspects of the serology kits. The tests were stratified into the following: report type (abstract, letter, or article), publication year (1991 to 1997), whether the study population was a consecutive series or a selection of a relevant study population, whether or not the patients had dyspeptic symptoms, the nationality of the study population, the reference standard used, the serology kit used, kit scale (quantitative, qualitative), the type of immunoglobulin (immunoglobulin A [IgA], IgG, and IgM simultaneously, IgA alone, or IgG alone) used to detect serum antibodies, the analysis technique of the serology kits (agglutination, enzyme immunoassay [EIA], enzyme-linked immunosorbent assay [ELISA], fixation, or immunochemical analysis), and whether whole blood or serum was used. We could not examine whether the generation of the test influenced performance because few studies mentioned this.

Statistical methods.

We first tried to model the heterogeneity between the studies by means of an ordinary least-squares regression equation, in which all the clinical features and methodological aspects were simultanously included. Unfortunately, this was not possible because of convergence problems. A best subset analysis was also not possible for the same reasons. Therefore, we decided to perform a separate regression analysis for each clinical feature. It is very likely that the AURCs for different serology kits are correlated when they are used with the same study population. By introducing a random effect for study population, we could model dependency between kits within the same study population (24) (see the Appendix). Moreover, the imprecision of the AURCs varied per study. In order to correct for the heterogeneity in the precision of the AURCs caused by different study sizes, we also performed a weighted regression analysis with weights proportional to 1/SE², where SE is the standard error (see the Appendix). Whenever AURC is equal to 1 or 0, SE will be 0. If this occurred, the study was excluded from the analysis.

For each regression model an overall F(NDF,DDF) test, where NDF is the degree of freedom in the numerator and DDF is the degree of freedom in the denominator of the F test, was used to examine whether the hypothesis β₁ = 0, β₂ = 0, . . ., β_k = 0 (no fixed effect) should be rejected. Akaike’s information criterion (AIC) is given to choose between the ordinary least-squares model, the random-effects model, and the weighted-random-effects model for each feature. The higher the AIC, the better the model fit. We also performed a weighted-random-effect regression analysis, using the significant features from the previous model simultaneously, in order to correct for possible confounding. These analyses were performed with SAS software (70). For multiple comparisons the Bonferroni correction was used to keep the overall α level at 0.05.

DATA SYNTHESIS

We found a total of 83 publications (1–23, 25–35, 37–41, 43–47, 49–69, 71–82, 85, 86, 88–91) with the MEDLINE search and by reviewing the references in the articles from the original search. Most publications had to be excluded because the H. pylori serology kit used was not commercially available. In those that could be included a total of 177 tests with 36 different commercially available H. pylori serology kits had been performed with 26,812 patients (Table 1). The medians (25 and 75% quantiles) of the sensitivity and specificity for H. pylori serology were 92% (85 and 96%) and 83% (73 and 92%), respectively. However, the sensitivities and specificities of the H. pylori serology kits ranged considerably between tests (Fig. 1).

TABLE 1.

H. pylori serology kits

Kit	Manufacturer	Technique	No. of populations	Size of total population (no. of subjects)	Sensitivity (%)a	Specificity (%)a	AURCa
μPlate	Boehringer	ELISA	1	102	93	92	0.90
AccuMeter	ChemTrak	Immunochemical	2	574	88	85	0.82
Anti-Hp	Roche	ELISA	18	3,558	94 (92–96)	84 (74–92)	0.85 (0.80–0.91)
Autoplate HF	Menarini	ELISA	1	70	91	96	0.89
Biolife	Biolife	ELISA	1	67	57	81	0.56
CFT H. pylori	Institute Virion	Fixation	2	1,040	77	74	0.73
Closer	Medical Instrument Corp.	EIA	1	86	96	73	0.83
Colorvue	Seradyn Clinical Diagnostics	EIA	1	80	93	88	0.88
Diesse	Senese	ELISA	2	136	96	99	0.96
Enzygnost	Behringwerke	ELISA	2	180	85	90	0.86
Flexpack	Abbott	Immunochemical	1	205	70	100	0.85
Flexsure	SmithKline	Immunochemical	12	1,837	93 (92–96)	75 (71–81)	0.84 (0.78–0.85)
GAP	Bio-Rad	ELISA	20	2,877	94 (88–97)	79 (65–90)	0.82 (0.68–0.88)
H. Pylori	Radim	ELISA	5	537	90 (84–91)	86 (83–90)	0.84 (0.80–0.85)
Hel-p	Amrad	ELISA	4	390	95 (93–98)	88 (83–91)	0.90 (0.86–0.92)
Helico-G	Porton	ELISA	16	2,328	89 (82–96)	75 (62–88)	0.77 (0.67–0.84)
Helisal	Cortecs	EIA and ELISA	5	754	82 (78–91)	83 (81–91)	0.79 (0.73–0.85)
Helori	Eurospital	ELISA	3	2,742	89 (81–96)	82 (68–96)	0.80 (0.77–0.88)
HM-CAP	Enteric Products	ELISA	8	1,216	94 (86–96)	83 (75–93)	0.86 (0.78–0.89)
Hp-G Screen	Genesis	ELISA	1	84	93	91	0.89
IIF Test	Bios	Immunochemical	1	110	98	31	0.65
ImmunoCard	Meridian Diagnostics	ELISA	2	188	94	87	0.88
Malakit	Biolab	ELISA	8	925	89 (85–96)	85 (74–96)	0.84 (0.76–0.92)
Meddens	Biotech	ELISA	1	102	98	95	0.96
Microstar	Kenstar	ELISA	1	84	97	76	0.86
Premier	Biomedical	ELISA	6	556	95 (89–100)	88 (81–96)	0.87 (0.85–0.90)
Pyloragen	Hycor Biomedical	ELISA	1	306	95	83	0.87
Pylori-Elisa	Bio-Whitaker	ELISA	1	84	100	96	0.98
Pylori-Stat	Whittaker	ELISA	14	1,243	96 (88–96)	88 (70–94)	0.85 (0.78–0.91)
Pyloriset	Orion	ELISA	15	1,807	90 (84–93)	89 (79–98)	0.84 (0.77–0.90)
Pyloriset latex	Orion	Agglutination	12	1,418	86 (64–92)	75 (66–91)	0.73 (0.60–0.85)
Quickvue	Quidel	EIA	3	456	92	88	0.78 (0.74–0.87)
Sia hp	Sigma	ELISA	1	84	90	98	0.89
SynElisa	Elias	ELISA	2	196	86	81	0.79
Trinity	Trinity Biotech	ELISA	2	206	89	78	0.80
Wampole	Carter-Wallace	ELISA	1	57	91	73	0.80

^aValues are medians (25 and 75% quantiles). 

FIG. 1.

Percent true-positive rates (TPR) and false-positive rates (FPR) for 177 tests.

Results for two tests were excluded from the regression analysis. Owing to their perfect diagnostic performance (the AURCs were 1) we could not calculate the SE of the AURC. According to the standard regression models, several clinical features and methodological aspects caused differences in diagnostic performance (Table 2). However, after correcting for the dependence between AURCs within the same study and the heterogeneity in the precision of the AURCs caused by different study sizes, only two aspects remained statistically significantly different. First, a major clinical feature of the study population that led to heterogeneity was the way in which the study population had been selected (Table 3). Second, because the investigated H. pylori serology kits were based on various antibody preparations, the diagnostic performances differed substantially. In the final weighted-random-effects regression model in which the features “consecutive yes/no” and “type of antibodies measured” were included, the estimated AURC for a nonconsecutive patient series was 0.053 (P = 0.01) higher than that for a consecutive patient series. The estimated AURC for kits that measured “IgA antibodies only” was 0.063 (P = 0.01) lower than that for kits that measured “IgG antibodies only,” while for kits that measured “IgA, IgG, and IgM simultaneously” it was 0.22 lower (P = <0.001). The P values for the overall F test in the multivariate weighted-random-effect regression analysis for the categories consecutive patients and antibodies were <0.001 and 0.013, respectively. The AURC for the serology kits that measured “IgA, IgG, and IgM antibodies” was 0.16 (P = 0.001) higher than that for the kits that measured “IgA antibodies alone.” After correcting for the way that the study population had been selected, an evaluation of only the serology kits that measured IgG antibodies with more than five test kits revealed that the diagnostic performance of the Helico-G serology kit was significantly lower (P = <0.001) than that of the Anti-Hp serology kit (Table 4). The overall F-test value for the serology kits with NDF equal to 8 and DDF equal to 47 was 4.07 (P = 0.001), and the overall F-test value for the consecutive category with NDF equal to 1 and DDF equal to 47 was 2.21 (P = 0.14).

TABLE 2.

Clinical features and methodological aspects of the selected studies associated with differences in test performance across studies

Analysis	Standard regression			Random-effect regressiona			Weighted-random-effect regressiona
	AICb	F(NDF,DDF)c	P value	AIC	F(NDF,DDF)	P value	AIC	F(NDF,DDF)	P value
Manuscript	121	F(2,174) = 1.10	0.33	138	F(2,96) = 0.44	0.64	144	F(2,95) = 0.40	0.67
Publication year	118	F(6,170) = 2.99	<0.01	130	F(6,95) = 0.77	0.59	135	F(6,94) = 0.87	0.52
Consecutive patients	130	F(1,175) = 15.9	<0.01	143	F(1,96) = 7.34	<0.01	148	F(1,95) = 7.36	<0.01
Patient population	122	F(2,174) = 2.17	0.12	138	F(2,96) = 0.55	0.58	144	F(2,95) = 0.55	0.58
Nationality	94	F(25,151) = 1.02	0.44	111	F(25,96) = 0.73	0.81	115	F(25,95) = 0.70	0.84
Reference standard	106	F(14,156) = 2.12	0.01	119	F(14,93) = 1.23	0.27	124	F(14,92) = 1.38	0.18
Serology kit	87	F(35,141) = 1.24	0.19	97	F(35,69) = 0.97	0.52	99	F(35,68) = 0.96	0.55
Kit scale	123	F(1,175) = 1.07	0.30	140	F(1,95) = 1.01	0.31	146	F(1,94) = 2.07	0.15
Antibody	135	F(2,169) = 21.5	<0.01	156	F(2,90) = 30.4	<0.01	161	F(2,89) = 29.4	<0.01
Kit technique	121	F(4,172) = 2.30	0.06	136	F(4,92) = 1.49	0.21	141	F(4,91) = 1.65	0.17
Blood or serum	123	F(1,175) = 0.23	0.63	140	F(1,95) = 0.59	0.45	146	F(1,94) = 1.55	0.22

^aRandom effect for study population. 

^bThe higher the AIC, the better the model fit. 

^cF(NDF,DDF), F test with degrees of freedom in the numerator (NDF) and denominator (DDF). 

TABLE 3.

Multivariate weighted-random-effect regression analysis for the statistically significant sources of heterogeneity^a

Source of heterogeneity	Overall test		Test for each category
	F(NDF,DDF)b	P value	Category	No. of populations	Size of population (no. of subjects)	Regression coefficient	SE	P value
Consecutive subjects	F(1,89) = 6.33	0.013	Yes	47	8,306	Reference
			No	130	18,379	0.053	0.021	0.01
Antibodiesc	F(2,89) = 28.71	<0.001	IgG	151	22,497	Reference
			IgA	7	1,272	−0.220	0.030	<0.01
			IgA, IgG, and IgM	14	2,458	−0.063	0.021	0.01

^aThe AIC value was 161, and the intercept was 0.806 (AURC for tests with a consecutive patient series and IgG antibody testing [reference categories]). 

^bSee footnote c of Table 2. 

^cUnknown for five tests. 

TABLE 4.

Difference in estimated AURCs in at least six studies that used serology kits measuring IgG antibodies, using weighted-regression analysis with random effects for study population, corrected for the way that the patient population was selected^a

	Difference in AURCs (P value)
	HM-CAP	Malakit	Pylori-Stat	Premier	GAP	Pyloriset	Flexsure	Helico-G
Anti-Hp	0.011 (0.62)	0.011 (0.61)	0.012 (0.53)	0.049 (0.08)	0.057 (0.004)	0.058 (0.01)	0.063 (0.01)	0.080 (<0.001)b
HM-CAP		<0.001 (0.98)	0.001 (0.93)	0.037 (0.16)	0.046 (0.06)	0.046 (0.05)	0.052 (0.006)	0.069 (0.005)
Malakit			<0.001 (0.97)	0.037 (0.24)	0.046 (0.07)	0.046 (0.07)	0.051 (0.07)	0.068 (0.008)
Pylori-Stat				0.036 (0.16)	0.044 (0.04)	0.045 (0.03)	0.051 (0.02)	0.067 (0.002)
Premier					0.007 (0.79)	0.008 (0.78)	0.014 (0.61)	0.030 (0.28)
GAP						<0.001 (0.99)	0.006 (0.80)	0.022 (0.26)
Pyloriset							0.006 (0.81)	0.022 (0.31)
Flexsure								0.016 (0.50)

^aThe AIC value was 110, and the overall F-test value for a serology kit with an NDF of 8 and a DDF of 47 was 4.07 (P = 0.001). 

^bStatistically significant according to Bonferroni’s correction [α¹ = (0.05/36) = 0.001). 

IMPLEMENTATION OF DIAGNOSTIC TESTS

Before implementing new diagnostic tests in clinical practice, careful evaluations must be done. Three topics are of importance (83). First, the test must have been evaluated with the indicated study population, i.e., the population of patients suspected of having the disease in question. The relevant population in this case consisted of a consecutive series of patients with dyspeptic complaints referred for upper gastrointestinal endoscopy. However, most studies analyzed a highly selected sample of patients with dyspepsia referred for upper gastrointestinal endoscopy. For a highly selected sample, H. pylori serology has excellent diagnostic performance. If a population of consecutive patients is tested, the diagnostic performance decreases.

The second topic of importance is determination of the diagnostic performance. For diagnostic tests, sensitivity and specificity are the most commonly used measures of test performance. Sensitivity and specificity are important parameters for diagnostic purposes but not for evaluative purposes. First, the use of different cutoff points for test positivity leads to various sensitivities and specificities. Second, the distribution of the test results for H. pylori-positive and -negative patients can vary considerably among studies because of selection. To overcome these problems, the presentation of the entire range of sensitivities and specificities at various cutoff points by a ROC curve results in better comparability of diagnostic tests. However, ROC curves and/or test result distributions were very sparsely presented in the publications. Fortunately, it is possible to make a fairly accurate estimation of the underlying ROC curve if one sensitivity and one specificity are mentioned (42). We think that this method is more appropriate for evaluation of the diagnostic performance of tests than the summary ROC technique used by Loy et al. (48). First, by analyzing the AURCs, the different cutoff points used for the same kit with different study populations were of no importance. Second, to compare pairs of serology kits, Loy et al. (48) needed more studies that had used two kits. By our method we could compare kits without any restrictions. A statistical test could be used to evaluate whether one was better than another. If the hypothesis of “no kit effect” is rejected, a multiple-comparison method can be used to compare pairs of tests. By introducing a random effect for study population as an alternative for the paired t test used by Loy et al. (48), we allowed for dependence between kits in the same study. Third, the weighted-regression method took the various study population sizes into account and corrected for them. Small studies with low AURCs will have smaller weights than large studies with high AURCs. The summary ROC analysis used equal weights for the studies involved, while it ignored the different study sizes and AURCs. Fourth, it seems to us that our method is likely to be more efficient for the testing of kits and other covariables because it incorporated studies and all kits in one analysis.

One problem remains: we do not know whether the random-effect model assumptions are correct. It is not clear whether the correlation between two kits and the correlation between two other kits within the same study population are equal. Maybe different correlations between different pairs of kits are better descriptions of reality. Unfortunately, most kits were used in only a few studies, and the number of kits used within one study was too small to thoroughly test the model assumptions. We used the MIXED procedure from the SAS software for the analysis. This procedure did not converge when all clinical features were included in the models because of too few observations. An automatic subsets analysis is not part of this SAS procedure. Therefore, we had to perform a univariate analysis. We could only perform a multivariate weighted-regression analysis with correction for random effects, in which the statistically significant clinical features from the univariate analysis were simultaneously evaluated in order to correct for possible confounding. A necessary condition for confounding is that the confounding variable is related to the feature under study and to the outcome (AURC). If a clinical feature or methodological aspect is not significant in a univariate model, then it is very unlikely to be significant in a model in which more features are included. However, the differences found by our method were confirmed in the analysis of kits and studies that fulfilled the requirements of Loy et al. (48). In agreement with Loy et al. (48), the Anti-Hp serology kit performed better than the Helico-G serology kit. Furthermore, the Malakit serology kit also displayed a higher although not statistically significant AURC than the Helico-G serology kit.

Finally, the relation between a new test and current diagnostic tests needs to be established. Many methods for the diagnosis of H. pylori infection are available. Because there is no consensus about a reference standard, several methods were used to identify H. pylori infection. The definition of the reference standard used in the publications ranged from only one diagnostic method (histology, culture, or rapid urease testing) having to be positive to more methods having to be positive (culture, rapid urease testing, and the urea breath test). The selection of a test as a reference reflects the personal preference of the investigator, which might lead to bias. Furthermore, the sensitivity and specificity of biopsy specimen-based methods vary and are frequently about 90%. Therefore, it is inappropriate to use other imperfect diagnostic tests as reference methods to measure diagnostic performance. However, the diagnostic performance of H. pylori serology was not influenced by any of the 15 different reference standards used.

RECOMMENDATION

In contrast to the conclusion drawn by Loy et al. (48) the diagnostic performances of various serology kits differed substantially because commercially available serology kits were based on various antibody preparations and were used with different study populations. Our results showed that serology kits that measured IgA, IgG, and IgM simultaneously (Pyloriset latex, CFT H. pylori) or IgA alone (Pyloriset, GAP) for the detection of H. pylori antibodies in serum did not perform as well as those that measured only IgG antibodies. The overall performance of commercially available serology kits that measure IgG antibodies for the diagnosis of H. pylori infection showed that serology is an accurate means of diagnosing H. pylori infection in patients. Owing to the small differences in diagnostic performance between serology kits that measure IgG antibodies, other aspects, such as the price, ease of handling, or number of equivocal results, are becoming increasingly important when choosing a serology kit.

Appendix

The regression equation used to model the heterogeneity between the studies by means of an ordinary least-squares regression analyses was as follows:

In this equation α is the intercept, β_j, j = 1,...,k, is the regression coefficient, and X_j, j = 1,...,k, is the dummy variable indicating the category of the clinical feature. The term ɛ is the residual which is normally distributed with variance ς_ɛ².

It is very likely that the AURCs for different serology kits are correlated when they are used within the same study population. By introducing a random effect for study population we could model dependency between kits within the same study population (24). The regression equation for the random effects model was as follows:

In this equation α, X_j, j = 1,...,k, and ɛ are as described above; β_j, j = 1,...,k, is now called the fixed effect; and b_l, l = 1,...,m, is the random effect which is independent and normally distributed with common variance ς_b². S_l, l = 1,...,m, is the dummy variable indicating the study population.

In order to correct for the heterogeneity in the precision of the AURCs caused by different study sizes, we also performed a weighted-regression analysis with weights proportional to 1/SE². The SE was computed according to the expression given by Hanley and McNeil (36):

where n_A is the number of abnormal individuals (H. pylori infected), and n_N is the number of healthy individuals (H. pylori noninfected). The expression for Q_l is given by Q_l = AURC/(2 − AURC), and that for Q₂ is given by Q₂ = (2 × AURC²)/(1 + AURC). This formula was derived under the assumption that the ratings are on a scale that is sufficiently continuous not to produce “ties”. Although in our case we used dichotomous tests, we believed that the formula for SE would nevertheless be useful. We used the SE only for the weighting procedure and expected to obtain the same answer when the unknown “true” SE was proportional to the SE that we used.