Abstract
Background. Multiple human papillomavirus (HPV) genotypes may be independently or synergistically associated with risk of high-grade squamous intraepithelial lesions (HSILs). We evaluated the risk of HSIL in women concomitantly infected with multiple HPV genotypes.
Methods. A population-based stratified sample of 59 664 cervical cytology specimens from women residing in New Mexico were evaluated for cytologic abnormalities and HPV genotypes. We calculated the risk of HSIL in women infected with a single HPV genotype and the risk in those infected with multiple HPV genotypes.
Results. The highest risk of HSIL was observed for HPV-16 (0.036), followed by HPV-33 (0.028), HPV-58 (0.024), and HPV-18 (0.022). For most types, we observed a greater risk of HSIL in women infected with multiple carcinogenic HPV types. In contrast, the risk of HSIL was similar in women infected with HPV-16 and other types, compared with women infected with HPV-16 only. We observed an increased but plateauing risk of HSIL in women infected with multiple types, compared with those infected with a single type, with risk ratios of 1.5 (95% confidence interval [CI], 1.2–1.8), 1.7 (95% CI, 1.3–2.4), and 1.4 (95% CI, 0.83–2.5) for women infected with 2, 3, and ≥4 genotypes, respectively.
Conclusions. In the largest population-based study of HPV genotypes and cytologic outcomes so far, we did not see more than additive effects of HPV types on the risk of HSIL in women infected with multiple types.
Keywords: cervical cancer screening, cytology, human papillomavirus (HPV), multiple Infections
Cervical cancer is caused by persistent infections with carcinogenic human papillomavirus (HPV) types. Although many of these infections are associated with minor cytomorphologic abnormalities, few progress to cervical precancer, and only a subset of precancers invade to become cancers [1].
The etiologic steps (infection, progression to precancer, invasion) of HPV-related carcinogenesis are well understood, but the determinants of progression remain largely unclear. Some behavioral factors, such as smoking, multiparity, and long-term hormonal contraceptive use, are associated with progression to cervical precancer and cancer, but the effects are modest [2]. In cohort studies, viral genotype has been found to be associated with progression [3–5]. HPV-16 is the most carcinogenic type and causes about 50% of cervical cancers worldwide, followed by HPV-18, HPV-45, and HPV-31 [6]. While several other types are commonly found among women without disease or with precancers, their attributable risks for cancer are much lower, compared with HPV-16 [7, 8].
Studying the risk of cancer precursors related to individual carcinogenic genotypes is complicated by the high prevalence of infection with multiple carcinogenic genotypes, especially among young women at the peak of sexual transmission of HPV [7, 9]. Biologically, there is no evidence for interaction between HPV genotypes with regard to the risk of cancer; therefore, it can be hypothesized that each infection is independently associated with a risk of precancer and cancer. However, there are controversial reports from cohort studies, some of them suggesting that HPV genotypes in infections involving multiple types act synergistically [10, 11]. Studying HPV genotype interactions is important to understand the biology of HPV infection and progression and to estimate the effect of vaccination on cervical disease outcomes.
Previous studies had limited numbers of disease end points. Here, we evaluated the associations between infection with single and multiple carcinogenic HPV genotypes and the risk of high-grade cytologic lesions in a large population-based study of >59 000 cervical cytology specimens.
METHODS
Population
The study sample was drawn from all 378 992 cervical cytology specimens collected between December 2007 and April 2009 from women residing in New Mexico and ascertained as previously described [12]. HPV genotyping was performed on a sample of 59 664 liquid-based cytology specimens from 7 in-state laboratories. A stratified random sample that used 4 strata based on age (≤30 vs >30 years) and cytology result (negative vs atypical squamous cells of undetermined significance [ASC-US] and worse) was used. Because the focus was the influence of HPV vaccination, there was oversampling of younger, cytology-negative women and women of all ages with abnormal cytology findings. Specimens were from 28 355 of 116 488 women (24.3%) aged ≤30 years with negative cytology results, 11 231 of 229 549 (4.9%) aged >30 years with negative cytology results, 11 155 of 18 962 (58.8%) aged ≤30 years with abnormal cytology results, and 8923 of 13 993 (63.8%) aged >30 years with abnormal cytology results (Table 1) were genotyped.
Table 1.
Cytology Results for Genotyped Samples and Sampling Fractions From the Overall Population
| Overall |
Negative |
ASC-US |
LSIL |
ASC-H |
HSIL |
AGC |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Age | No. (%) | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % | Proportion (%)a | Sampling Fraction, % |
| ≤30 y | 39 490 (100) | 29.2 | 28 335/39 490 (71.8) | 24.3 | 6048/39 490 (15.3) | 59.3 | 3914/39 490 (9.9) | 56.8 | 551/39 490 (1.4) | 65.4 | 490/39 490 (1.2) | 61.0 | 152/39 490 (0.4) | 65.8 |
| >30 y | 20 154 (100) | 8.3 | 11 231/20 154 (55.7) | 4.9 | 5960/20 154 (29.6) | 64.3 | 1704/20 154 (8.5) | 62.8 | 431/20 154 (2.1) | 66.0 | 280/20 154 (1.4) | 54.4 | 548/20 154 (2.7) | 65.4 |
Abbreviations: AGC, atypical glandular cells; ASC-H, atypical squamous cells, rule out HSIL; ASC-US, atypical squamous cells of undetermined significance; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.
a Data are no. of specimens with the characteristic/no. tested (%).
The research reported here was interfaced to the New Mexico HPV Pap Registry (NMHPVPR) and was approved by the University of New Mexico Human Research Review Committee. The NMHPVPR was established to monitor the continuum of cervical screening preventive care and to characterize the population-level influence of HPV vaccination through surveillance and approved research investigations.
HPV Genotyping
The LINEAR ARRAY HPV Genotyping Test (HPV LA; Roche Diagnostics, Indianapolis, IN) is a nonquantitative test for 37 HPV genotypes that incorporates selective polymerase chain reaction (PCR) amplification with biotinylated PGMY 09/11 L1 region consensus primers and colorimetric detection of amplified products bound to immobilized HPV genotype–related oligonucleotide probes on a LINEAR ARRAY HPV genotyping strip. PGMY-based HPV genotyping with the HPV LA and prototype Line Blot assay was conducted in SurePath (Becton-Dickinson, Franklin Lakes, NJ) or ThinPrep (Hologic, Boxborough, MA) specimens and has been previously reported in detail for this population [12]. Both SurePath and ThinPrep samples were refrigerated after collection and were processed between 30 and 45 days (SurePath) or between 45 days and 6 months (ThinPrep), respectively, following clinical collection. After vigorous mixing of the original liquid cytology specimens, 500-µL aliquots of SurePath or ThinPrep solution were transferred to 12 mm × 75 mm polypropylene tubes, and DNA was purified using a Cobas X421 robot (Roche Molecular Systems [RMS], Pleasanton, CA). The robot performed proteinase K digestion and inactivation with the final DNA eluate (150 µL) delivered into a 96-well QiaAmp plate (Qiagen, Valencia, CA). Fifty microliters (50 µL) of purified DNA was transferred to a tube with 50 µL of HPV Linear Array master mix, and the mixture was amplified by PCR, using the Applied Biosystems (Foster City, CA) gold-plated 96-well GeneAmp PCR System 9700 as specified by the manufacturer. Controls for contamination and assay sensitivity were included in each 96-well assay.
By use of the Roche HPV LA detection kit, hybridizations were automated using Tecan ProfiBlot-48 robots (Tecan, Grödig, Austria). The Roche HPV LA Genotyping Test detects 13 high-risk and 24 low-risk HPV types. HPV-52 is not determined directly by a type-specific probe but rather by a probe that cross-hybridizes with HPV-33, -35, -52, and -58. The presence of HPV-52 was inferred only if the cross-reactive probe was hybridized but no hybridization was detected for the probes specific for HPV-33, -35, and -58. Notably, concurrent infections involving HPV-52 and any of the 3 other types cannot be detected. Two independent readers interpreted the presence of HPV genotypes, using a reference template provided by the manufacturer. Any discrepancies were identified by a custom computer application applied to the data input and were adjudicated by a third review. The present analysis was restricted to the following 13 carcinogenic types: HPV-16, -18, -31, -33, -35, -39, -45, -51, -52, -56, -58, -59, and -68.
Cervical Cytology
Liquid-based cytology was performed in 7 laboratories, using either ThinPrep (Hologic) or SurePath (BD) liquid-based cytology systems. Cytology results were classified according to the 2001 Bethesda System (TBS): high-grade squamous intraepithelial lesion (HSIL), atypical squamous cells cannot rule out HSIL (ASC-H), atypical glandular cells (AGC), low-grade squamous intraepithelial lesion (LSIL), atypical squamous cells of undetermined significance (ASC-US), and negative for intraepithelial lesion or malignancy. Cytology results reported as cervical intraepithelial neoplasia (CIN) grade 1 were classified as LSIL, and results reported as CIN grade 2 (CIN2), CIN grade 3 (CIN3), carcinoma in situ (CIS), or possible carcinoma were classified as HSIL. Cytology results were based on local readings, and no attempt was made to review them centrally or undertake quality assurance activities.
Statistical Analysis
We calculated proportions of cytology results given the presence of specific genotypes after weighting back to the whole population by using the sampling fractions shown in Table 1. We stratified the analyses by age (≤30 and >30 years), by infection with single or multiple types, and in infections with multiple types, by whether HPV-16 was present or not. We performed χ2 tests to evaluate whether the risk of any cytologic abnormality and of HSIL differed across all carcinogenic genotypes present in single infections. Next, for each individual genotype, we performed chi-square tests, comparing the proportion of ASC-US or greater and of HSIL by presence of single or multiple infections, and in multiple infections, whether HPV16 was present or not. We evaluated the influence of the number of carcinogenic types on the risk of cytologic abnormalities by analyzing proportions of cytology categories related to the number of carcinogenic HPV types in the complete sample and after excluding women with HPV-16 infections. Raw P values are presented in the text and tables, with footnotes indicating P values that were statistically significant after conservative Bonferroni correction (P < .0038). This presentation permits an assessment of the results without adjustment for multiple comparisons and with the most conservative correction for multiple comparisons.
We ran logistic regression models for the risk of HSIL versus the risk of other cytologic findings, stratified by the number of HPV genotypes detected, and we repeated the runs after excluding women infected with HPV-16. We ran weighted χ2 tests to examine the role of multiple HPV genotypes, with or without HPV-16, on the risk of HSIL. First, in a model that included all women with at least 1 HPV type, we computed a P value from a χ2 test for the difference in HSIL between women with 1 versus >1 type. Next, we limited the analysis to women who had multiple (ie, ≥2) types and repeated the analysis to see whether there was any difference between women who had 2 types and those who had ≥3 types. Furthermore, we also compared 2–3 types with ≥4 types. All analyses were run in SAS, version 9.1.
RESULTS
Risk of Abnormal Cytology Results, by HPV Genotype
As the first fundamental observation, for all HPV types, most associated cytology results were negative (Table 2). Thus, most infections were not accompanied by even equivocal cytologic abnormalities (ASC-US), as diagnosed in routine clinical practice. HPV-16 was the most common carcinogenic HPV genotype in the stratified sample (n = 4234), followed by HPV-51 (n = 2878), HPV-39 (n = 2498), HPV-59 (n = 2275), and HPV-31 (n = 2122; Table 2). The least common carcinogenic types in the sample were HPV-45 (n = 1130), HPV-35 (n = 1024), HPV-68 (n = 815), and HPV-33 (n = 591).
Table 2.
Proportion of Cytology Results, by Human Papillomavirus (HPV) Genotype, Weighted to the Overall Population
| Type, Age | Specimens, No. (%) | Weighted Proportion |
||||
|---|---|---|---|---|---|---|
| NILM | ASC-US | LSIL | ASC-H | HSIL | ||
| HPV-16 | ||||||
| All | 4234 | 0.678 | 0.145 | 0.111 | 0.030 | 0.036 |
| ≤30 y | 3473 (82) | 0.632 | 0.164 | 0.136 | 0.033 | 0.036 |
| >30 y | 761 (18) | 0.779 | 0.104 | 0.057 | 0.024 | 0.036 |
| HPV-18 | ||||||
| All | 1461 | 0.689 | 0.155 | 0.115 | 0.018 | 0.022 |
| ≤30 y | 1105 (76) | 0.650 | 0.173 | 0.138 | 0.019 | 0.020 |
| >30 y | 356 (24) | 0.765 | 0.120 | 0.070 | 0.018 | 0.027 |
| HPV-31 | ||||||
| All | 2122 | 0.703 | 0.153 | 0.099 | 0.025 | 0.020 |
| ≤30 y | 1629 (77) | 0.655 | 0.178 | 0.119 | 0.025 | 0.023 |
| >30 y | 493 (23) | 0.793 | 0.106 | 0.060 | 0.026 | 0.015 |
| HPV-33 | ||||||
| All | 591 | 0.673 | 0.152 | 0.122 | 0.024 | 0.028 |
| ≤30 y | 445 (75) | 0.630 | 0.167 | 0.153 | 0.018 | 0.033 |
| >30 y | 146 (25) | 0.754 | 0.125 | 0.065 | 0.036 | 0.019 |
| HPV-35 | ||||||
| All | 1024 | 0.685 | 0.152 | 0.122 | 0.020 | 0.021 |
| ≤30 y | 734 (72) | 0.612 | 0.182 | 0.158 | 0.021 | 0.026 |
| >30 y | 290 (28) | 0.786 | 0.110 | 0.072 | 0.018 | 0.014 |
| HPV-39 | ||||||
| All | 2498 | 0.717 | 0.141 | 0.120 | 0.012 | 0.009 |
| ≤30 y | 2037 (82) | 0.672 | 0.161 | 0.144 | 0.013 | 0.010 |
| >30 y | 461 (18) | 0.824 | 0.094 | 0.066 | 0.010 | 0.006 |
| HPV-45 | ||||||
| All | 1130 | 0.750 | 0.135 | 0.091 | 0.015 | 0.008 |
| ≤30 y | 832 (74) | 0.708 | 0.159 | 0.108 | 0.018 | 0.008 |
| >30 y | 298 (26) | 0.818 | 0.098 | 0.065 | 0.010 | 0.009 |
| HPV-51 | ||||||
| All | 2878 | 0.656 | 0.150 | 0.159 | 0.018 | 0.016 |
| ≤30 y | 2339 (81) | 0.605 | 0.168 | 0.188 | 0.021 | 0.018 |
| >30 y | 539 (19) | 0.780 | 0.106 | 0.089 | 0.012 | 0.013 |
| HPV-52 | ||||||
| All | 2035 | 0.763 | 0.132 | 0.078 | 0.016 | 0.011 |
| ≤30 y | 1566 (77) | 0.726 | 0.154 | 0.093 | 0.014 | 0.013 |
| >30 y | 469 (23) | 0.831 | 0.091 | 0.052 | 0.020 | 0.007 |
| HPV-56 | ||||||
| All | 1709 | 0.608 | 0.161 | 0.209 | 0.012 | 0.010 |
| ≤30 y | 1240 (73) | 0.565 | 0.175 | 0.238 | 0.012 | 0.010 |
| >30 y | 469 (27) | 0.686 | 0.137 | 0.156 | 0.013 | 0.008 |
| HPV-58 | ||||||
| All | 1532 | 0.673 | 0.160 | 0.123 | 0.020 | 0.024 |
| ≤30 y | 1215 (79) | 0.641 | 0.173 | 0.140 | 0.021 | 0.025 |
| >30 y | 317 (21) | 0.750 | 0.128 | 0.083 | 0.018 | 0.021 |
| HPV-59 | ||||||
| All | 2275 | 0.759 | 0.132 | 0.094 | 0.008 | 0.007 |
| ≤30 y | 1855 (82) | 0.713 | 0.154 | 0.116 | 0.009 | 0.007 |
| >30 y | 420 (18) | 0.855 | 0.086 | 0.048 | 0.005 | 0.006 |
| HPV-68 | ||||||
| All | 815 | 0.758 | 0.130 | 0.094 | 0.014 | 0.005 |
| ≤30 y | 606 (74) | 0.676 | 0.171 | 0.129 | 0.020 | 0.005 |
| >30 y | 209 (26) | 0.863 | 0.078 | 0.049 | 0.006 | 0.005 |
Abbreviations: ASC-H, atypical squamous cells, rule out HSIL; ASC-US, atypical squamous cells of undetermined significance; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.
For each type (whether occurring alone or with other genotypes), we calculated the proportions of associated cytology results. The highest (weighted) proportions associated with HSIL were observed for HPV-16 (0.036), followed by HPV-33 (0.028), HPV-58 (0.024), HPV-18 (0.022), HPV-35 (0.021), and HPV-31 (0.020; Table 2). Importantly, these are not the fractions of HSIL cases associated with each type but, rather, the fractions of the infections associated with HSIL. HPV-56 and HPV-51 had the highest proportions of LSIL (0.209 and 0.159, respectively). We stratified the population by age, using ≤30 years and >30 years, and again evaluated the proportions of cytology results related to individual genotypes (Table 2). For all genotypes, among women aged >30 years, we observed a higher proportion of negative cytology results and a much lower proportion of LSIL cytology results, compared with the younger group. In contrast, the difference in HSIL risk between the 2 age categories was less consistent: some genotypes (eg, HPV-33 and HPV-51) were associated with a higher risk of HSIL in the younger group, whereas others (eg, HPV-18 and HPV-45) had a higher risk in the older group.
We examined the relationship of HPV type to cytologic abnormality in women infected with a single HPV genotype. Overall, the cytology result (ie, any abnormality vs a negative finding, or HSIL vs a finding less severe than HSIL) differed significantly by HPV genotype (P < .001), based on the distribution of cytology findings in the first row for each genotype in Table 3. The same heterogeneity was observed when this relationship was analyzed individually in the younger and older age groups (P < .001; Supplementary Tables 1 and 2).
Table 3.
Cytology Results in Infections Involving Single and Multiple Human Papillomavirus (HPV) Genotypes Among All Sampled Women
| Type | Specimens, No. | Specimens, Proportion |
Pa |
|||||
|---|---|---|---|---|---|---|---|---|
| NILM | ASC-US | LSIL | ASC-H | HSIL | All | HSIL | ||
| HPV-16 | ||||||||
| Only | 2133 | 0.754 | 0.114 | 0.068 | 0.028 | 0.037 | <.00001b | .68955 |
| Plus others | 2101 | 0.571 | 0.189 | 0.172 | 0.033 | 0.035 | ||
| HPV-18 | ||||||||
| Only | 596 | 0.789 | 0.105 | 0.077 | 0.011 | 0.018 | <.00001b | .11492 |
| Plus others | ||||||||
| Overall | 834 | 0.582 | 0.208 | 0.156 | 0.026 | 0.027 | ||
| No HPV-16 | 574 | 0.589 | 0.206 | 0.16 | 0.023 | 0.021 | .38391 | |
| Includes HPV-16 | 260 | 0.566 | 0.212 | 0.148 | 0.034 | 0.04 | ||
| HPV-31 | ||||||||
| Only | 982 | 0.787 | 0.121 | 0.058 | 0.02 | 0.014 | <.00001b | .00255b |
| Plus others | ||||||||
| Overall | 1119 | 0.595 | 0.195 | 0.151 | 0.032 | 0.027 | ||
| No HPV-16 | 756 | 0.618 | 0.184 | 0.148 | 0.025 | 0.025 | .0561 | |
| Includes HPV-16 | 363 | 0.545 | 0.22 | 0.156 | 0.046 | 0.034 | ||
| HPV-33 | ||||||||
| Only | 259 | 0.781 | 0.1 | 0.077 | 0.023 | 0.019 | <.00001b | .03466 |
| Plus others | ||||||||
| Overall | 325 | 0.533 | 0.219 | 0.18 | 0.026 | 0.041 | ||
| No HPV-16 | 208 | 0.575 | 0.225 | 0.152 | 0.018 | 0.03 | .02757 | |
| Includes HPV-16 | 117 | 0.447 | 0.207 | 0.24 | 0.044 | 0.062 | ||
| HPV-35 | ||||||||
| Only | 459 | 0.747 | 0.14 | 0.082 | 0.014 | 0.017 | <.00001b | .23078 |
| Plus others | ||||||||
| Overall | 558 | 0.618 | 0.165 | 0.166 | 0.026 | 0.025 | ||
| No HPV-16 | 394 | 0.631 | 0.156 | 0.167 | 0.024 | 0.022 | .7094 | |
| Includes HPV-16 | 164 | 0.584 | 0.188 | 0.165 | 0.032 | 0.032 | ||
| HPV-39 | ||||||||
| Only | 1085 | 0.818 | 0.097 | 0.077 | 0.004 | 0.005 | <.00001b | .00036b |
| Plus others | ||||||||
| Overall | 1396 | 0.599 | 0.192 | 0.172 | 0.023 | 0.014 | ||
| No HPV-16 | 997 | 0.626 | 0.181 | 0.168 | 0.018 | 0.008 | .00001b | |
| Includes HPV-16 | 399 | 0.524 | 0.225 | 0.183 | 0.035 | 0.033 | ||
| HPV-45 | ||||||||
| Only | 447 | 0.878 | 0.071 | 0.039 | 0.007 | 0.004 | <.00001b | .01106 |
| Plus others | ||||||||
| Overall | 665 | 0.592 | 0.216 | 0.155 | 0.024 | 0.014 | ||
| No HPV-16 | 479 | 0.619 | 0.202 | 0.15 | 0.017 | 0.012 | .03944 | |
| Includes HPV-16 | 186 | 0.512 | 0.256 | 0.172 | 0.043 | 0.017 | ||
| HPV-51 | ||||||||
| Only | 1268 | 0.744 | 0.108 | 0.125 | 0.011 | 0.012 | <.00001b | .00753 |
| Plus others | ||||||||
| Overall | 1590 | 0.56 | 0.196 | 0.197 | 0.026 | 0.021 | ||
| No HPV-16 | 1120 | 0.58 | 0.19 | 0.189 | 0.022 | 0.018 | .03521 | |
| Includes HPV-16 | 470 | 0.509 | 0.21 | 0.216 | 0.036 | 0.03 | ||
| HPV-52 | ||||||||
| Only | 942 | 0.844 | 0.097 | 0.042 | 0.011 | 0.005 | <.00001b | .00013b |
| Plus others | ||||||||
| Overall | 1073 | 0.655 | 0.178 | 0.126 | 0.023 | 0.017 | ||
| No HPV-16 | 720 | 0.683 | 0.177 | 0.114 | 0.018 | 0.009 | .00003b | |
| Includes HPV-16 | 353 | 0.59 | 0.182 | 0.155 | 0.037 | 0.037 | ||
| HPV-56 | ||||||||
| Only | 736 | 0.687 | 0.128 | 0.174 | 0.006 | 0.006 | <.00001b | .0283 |
| Plus others | ||||||||
| Overall | 961 | 0.526 | 0.196 | 0.245 | 0.02 | 0.014 | ||
| No HPV-16 | 688 | 0.541 | 0.198 | 0.233 | 0.018 | 0.011 | .31507 | |
| Includes HPV-16 | 273 | 0.485 | 0.193 | 0.279 | 0.023 | 0.02 | ||
| HPV-58 | ||||||||
| Only | 696 | 0.741 | 0.146 | 0.078 | 0.015 | 0.02 | <.00001b | .26384 |
| Plus others | ||||||||
| Overall | 827 | 0.6 | 0.175 | 0.172 | 0.026 | 0.027 | ||
| No HPV-16 | 567 | 0.631 | 0.172 | 0.152 | 0.024 | 0.021 | .00492 | |
| Includes HPV-16 | 260 | 0.52 | 0.184 | 0.224 | 0.031 | 0.042 | ||
| HPV-59 | ||||||||
| Only | 966 | 0.862 | 0.09 | 0.043 | 0.003 | 0.002 | <.00001b | <.00001b |
| Plus others | ||||||||
| Overall | 1300 | 0.64 | 0.181 | 0.153 | 0.013 | 0.013 | ||
| No HPV-16 | 923 | 0.668 | 0.171 | 0.143 | 0.011 | 0.006 | .00001b | |
| Includes HPV-16 | 377 | 0.559 | 0.21 | 0.182 | 0.019 | 0.031 | ||
| HPV-68 | ||||||||
| Only | 304 | 0.874 | 0.072 | 0.051 | 0.003 | 0 | <.00001b | .00221b |
| Plus others | ||||||||
| Overall | 503 | 0.628 | 0.194 | 0.142 | 0.026 | 0.01 | ||
| No HPV-16 | 365 | 0.657 | 0.179 | 0.128 | 0.028 | 0.007 | .07408 | |
| Includes HPV-16 | 138 | 0.542 | 0.24 | 0.181 | 0.018 | 0.018 | ||
Abbreviations: ASC-H, atypical squamous cells, rule out HSIL; ASC-US, atypical squamous cells of undetermined significance; CI, confidence interval; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.
a The column titled “All” contains results of comparisons of all cytology results, and the column titled “HSIL” contains results of comparisons of HSIL with findings less severe than HSIL. From the same table, P < .0001 for comparison of HSIL with findings less severe than HSIL across all genotypes present in single carcinogenic HPV infections (P < .0001). All P values were calculated by the χ2 test.
b Statistically significant at a Bonferroni threshold (P < .0038).
Effect of Coinfections on the Risk of Abnormal Cytology Results for Individual Genotypes
For HPV-16, the risk of HSIL was slightly but not significantly higher in women infected with HPV-16 only, compared with women infected with HPV-16 and ≥1 other carcinogenic type. For all other types, we observed a greater risk of HSIL in women infected with multiple carcinogenic HPV types that was significant for HPV-31 (P = .003, significant at the Bonferroni threshold), HPV-33 (P = .03), HPV-39 (P = .0004, significant at the Bonferroni threshold), HPV-45 (P = .01), HPV-51 (P = .008), HPV-52 (P = .0001, significant at the Bonferroni threshold), HPV-56 (P = .03), HPV-59 (P < .0001, significant at the Bonferroni threshold), and HPV-68 (P < .002, significant at the Bonferroni threshold; Table 3). When comparing infections with multiple types that included HPV-16 with those that did not, all infections with multiple types were shifted to higher cytology categories when they included HPV-16, compared with the multiple-type infections without HPV-16 (for HPV-33, P = .03; for HPV-39, P < .0001, significant at the Bonferroni threshold; for HPV-45, P = .04; for HPV-52, P < .0001, significant at the Bonferroni threshold; for HPV-58, P = .005; and for HPV-59, P < .0001, significant at the Bonferroni threshold), but for all types, the risk of HSIL for infection with multiple types other than HPV-16 was still higher than that for infection with a single type. For all genotypes, we observed a higher risk of LSIL when the genotype was present with other genotypes, compared with when it occurred alone. Notably, for all types except HPV-18 and HPV-35 that were present in women infected with multiple types, the risk of LSIL was higher in women in whom infection included HPV-16 (Table 3).
For each HPV type, in women ≤30 years old coinfection with another type was more common than monoinfection (Supplementary Table 1), whereas in women >30 years old monoinfection was more common than coinfection with another type (Supplementary Table 2). The risk of LSIL was much lower for all genotypes among older women, compared with that for younger women.
For HPV-16, the risk of HSIL among women aged ≤30 years infected with HPV-16 alone was nonsignificantly greater than the risk for those infected with HPV-16 and ≥1 other carcinogenic type (P = .14; Supplementary Table 1), whereas the risk among women aged >30 years infected with HPV-16 alone was nonsignificantly less than the risk for those infected with HPV-16 and ≥1 other carcinogenic type (P = .09; Supplementary Table 2). For most other types, the effects were similar in age-stratified analyses, compared with the combined analysis. Consistently for all types, the risk of HSIL was highest for women in whom infection included HPV-16 and ≥1 other type.
Infection With Multiple Carcinogenic HPV Types and the Risk of Abnormal Cytology Results
We analyzed the relationship between the risk of abnormal cytology results and infection with increasing numbers of carcinogenic genotypes. In women infected with multiple types, we observed an increasing proportion with any cytologic abnormality and with HSIL (Table 4). Compared with the risk of HSIL among women infected with a single HPV type, the risk of HSIL among women infected with 2 types was 1.5-fold greater (95% confidence interval [CI], 1.2–1.8), the risk among women infected with 3 types was 1.7-fold greater (95% CI, 1.3–2.4), and the risk among women infected with ≥4 types was 1.4-fold greater (95% CI, .83–2.5). Although we observed a significant increase in the risk of HSIL among women infected with ≥2 types, compared with those infected with 1 type (P < .0001), we did not see a significant increase between women infected with ≥3 types and those infected with 2 types or between women infected with ≥4 types and those infected with 2 or 3 types (P>.05). In age-stratified analyses, slightly higher risk ratios were observed for women >30 years of age. To evaluate type multiplicity without the effects of HPV-16 in multiple genotype combinations, we repeated the analysis after excluding all women with HPV-16 infections (Table 5). While the absolute risk of HSIL was lower in HPV-16-negative women (eg, 0.014 for women with 2 carcinogenic types but without HPV-16 vs 0.021 for women with 2 carcinogenic types including HPV-16), the increase in relative risk for women infected with multiple types was very similar to what was observed when HPV-16 was included in the analysis, with risk ratios of 1.5 (95% CI, 1.1–2.0) in women infected with 2 types, 1.7 (95% CI, 1.0–2.9) in women infected with 3 types, and 1.6 (95% CI, .60–4.5) in women infected with ≥4 types. Similar to the overall analysis, there was a significant increase in risk among women infected with ≥2 types, compared with those infected with 1 type (P = .0007), but we observed no further increase in risk for additional genotypes.
Table 4.
Number of Carcinogenic Human Papillomavirus (HPV) Types and Risk of Cytologic Abnormalities for All Sampled Women
| No. of Types, Age | Specimens, No. | Specimens, Proportion |
HSIL |
||||
|---|---|---|---|---|---|---|---|
| NILM | ASC-US | LSIL | ASC-H | Specimens, Proportion | ORa (95% CI) | ||
| 1 | |||||||
| All | 10 873 | 0.790 | 0.107 | 0.075 | 0.013 | 0.014 | 1.0 |
| ≤30 y | 8113 | 0.748 | 0.125 | 0.098 | 0.013 | 0.015 | 1.0 |
| >30 y | 2760 | 0.843 | 0.085 | 0.046 | 0.013 | 0.013 | 1.0 |
| 2 | |||||||
| All | 3743 | 0.644 | 0.170 | 0.143 | 0.022 | 0.021 | 1.5 (1.2–1.8) |
| ≤30 y | 3141 | 0.625 | 0.179 | 0.152 | 0.023 | 0.021 | 1.4 (1.1–1.8) |
| >30 y | 602 | 0.705 | 0.139 | 0.116 | 0.020 | 0.020 | 1.6 (1.1–2.4) |
| 3 | |||||||
| All | 1211 | 0.538 | 0.213 | 0.196 | 0.028 | 0.024 | 1.7 (1.3–2.4) |
| ≤30 y | 1074 | 0.522 | 0.222 | 0.207 | 0.025 | 0.025 | 1.6 (1.1–2.3) |
| >30 y | 147 | 0.615 | 0.174 | 0.144 | 0.044 | 0.024 | 1.8 (.84–4.0) |
| ≥4 | |||||||
| All | 490 | 0.481 | 0.236 | 0.231 | 0.032 | 0.020 | 1.4 (.83–2.5) |
| ≤30 y | 452 | 0.486 | 0.236 | 0.226 | 0.033 | 0.019 | 1.2 (.70–2.3) |
| >30 y | 38 | 0.420 | 0.242 | 0.290 | 0.016 | 0.032 | 2.6 (.55–12) |
Abbreviations: ASC-H, atypical squamous cells, rule out HSIL; ASC-US, atypical squamous cells of undetermined significance; CI, confidence interval; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.
a The age-specific risk for infection with 1 carcinogenic HPV genotype were the reference values used to calculate the age-specific risk ratios (ORs) for infections with >1 carcinogenic HPV genotype. P < .0001, by χ2 analysis, for the risk of HSIL in women infected with 1 vs those infected with ≥2 genotypes; P > .05, by χ2 analysis, for the risk of HSIL in women infected with 2 vs those infected with >2 genotypes.
Table 5.
Number of Carcinogenic Human Papillomavirus (HPV) Types and Risk of Cytologic Abnormalities, Excluding Women Infected With HPV-16
| No. of Types, Age | Specimens, No. | Specimens, Proportion |
HSIL |
||||
|---|---|---|---|---|---|---|---|
| NILM | ASC-US | LSIL | ASC-H | Specimens, Proportion | ORa (95% CI) | ||
| 1 | |||||||
| All | 8740 | 0.799 | 0.106 | 0.077 | 0.010 | 0.009 | 1.0 |
| ≤30 y | 6501 | 0.758 | 0.123 | 0.100 | 0.009 | 0.009 | 1.0 |
| >30 y | 2239 | 0.849 | 0.084 | 0.047 | 0.011 | 0.009 | 1.0 |
| 2 | |||||||
| All | 2491 | 0.657 | 0.172 | 0.139 | 0.018 | 0.014 | 1.5 (1.1–2.0) |
| ≤30 y | 2049 | 0.640 | 0.183 | 0.145 | 0.017 | 0.015 | 1.6 (1.2–2.3) |
| >30 y | 442 | 0.707 | 0.142 | 0.121 | 0.020 | 0.010 | 1.2 (.63–2.1) |
| 3 | |||||||
| All | 679 | 0.553 | 0.212 | 0.199 | 0.020 | 0.016 | 1.7 (1.0–2.9) |
| ≤30 y | 587 | 0.525 | 0.230 | 0.211 | 0.015 | 0.019 | 2.0 (1.2–3.4) |
| >30 y | 92 | 0.662 | 0.144 | 0.152 | 0.038 | 0.004 | 0.5 (.06–3.5) |
| ≥4 | |||||||
| All | 183 | 0.525 | 0.196 | 0.226 | 0.038 | 0.015 | 1.6 (.60–4.5) |
| ≤30 y | 170 | 0.550 | 0.187 | 0.211 | 0.040 | 0.012 | 1.3 (.40–4.0) |
| >30 y | 13 | 0.000 | 0.385 | 0.538 | 0.000 | 0.077 | 9.4 (1.2–73) |
Abbreviations: ASC-H, atypical squamous cells, rule out HSIL; ASC-US, atypical squamous cells of undetermined significance; CI, confidence interval; HSIL, high-grade squamous intraepithelial lesion; LSIL, low-grade squamous intraepithelial lesion; NILM, negative for intraepithelial lesion or malignancy.
a The age-specific risk for infection with 1 carcinogenic HPV genotype were the reference values used to calculate the age-specific risk ratios (ORs) for infections with >1 carcinogenic HPV genotype. P = .0007, by χ2 analysis, for the risk of HSIL in women infected with 1 vs those infected with ≥2 genotypes; P > .05, by χ2 analysis, for the risk of HSIL in women infected with 2 vs those infected with >2 genotypes.
DISCUSSION
Concomitant infection with multiple carcinogenic HPV types is very common, especially among younger women [7]. The effects of concurrent infection with multiple HPV types on cytologic changes and the risk of precancer and cancer are not well understood. It is widely assumed that each HPV genotype acts independently and, thus, should independently contribute to risk of cytologic changes and precancer. To support this notion, it has been demonstrated that individual histologic cervical lesions are usually caused by a single genotype, even if multiple types are present in the cytology specimen [13, 14]. Under this assumption, we would expect to see that concomitant infection with multiple types yields additive effects on the risk of abnormal cytology findings and cervical precancer. However, some previous studies have suggested that infection with multiple HPV types yields synergistic effects [11], whereas others have not [10].
In the largest population-based study of disease outcomes in women with HPV genotyping information reported to date, we evaluated the risk of cytologic abnormalities associated with individual genotypes and the role of infection with multiple carcinogenic HPV types.
We demonstrated that most HPV infections do not present in routine practice with even equivocal cytologic abnormalities, confirming previous reports [15, 16]. For all types and type combinations, ASC-US was the most common cytologic abnormality, followed by LSIL, ASC-H, and HSIL. However, the proportion of cytologic abnormalities strongly varied by HPV type, both overall and when analyses were restricted to infection with single carcinogenic types. HPV-16 was most strongly associated with HSIL, whereas HPV-56 was most strongly associated with LSIL. Our findings confirm that HPV-16 is the most dominant etiological type, showing the highest risk association with HSIL, compared with all other genotypes in single infections, and showing an increasing risk for HSIL in all type combinations, compared with other coinfections. These findings support recent data from tissue-based genotyping, showing that when HPV-16 is present with another carcinogenic type in cytology samples, HPV-16 is usually the causal type in cervical lesions [14].
In our study, women infected with multiple carcinogenic HPV types had a much higher risk of having any cytologic abnormality. Infection with increasing numbers of HPV genotypes had the strongest effect on the LSIL category, with up to a 3-fold increased risk (LSIL proportion, 0.075 among women infected with 1 type and 0.231 among women infected with ≥4 types). Women infected with multiple concomitant carcinogenic types also had a higher risk of HSIL, but the risk did not further increase beyond concomitant infection with 2 types (HSIL proportion, 0.014 among women infected with 1 genotype and around 0.021 among women infected with 2, 3, or ≥4 types). Notably, for women infected with HPV-16, we observed that the risk of HSIL was unchanged when HPV-16 was present in a coinfection. Under the assumption of type independence, we would have expected a higher risk of HSIL in women infected with multiple types. However, we observed a strong increase in LSIL results among women infected with HPV-16 plus other carcinogenic types. It is conceivable that a cytology finding with a strong LSIL pattern can mask the presence of few abnormal cells that would be recognized more easily without the background of viral changes caused by multiple infections.
Overall, we did not see evidence of a more-than-additive effect of the presence of multiple carcinogenic genotypes on high-grade cytology results. In fact, our estimates for the risk of HSIL related to infection with >2 carcinogenic types did not increase beyond the risk observed for infection with 2 carcinogenic genotypes. As outlined above for HPV-16, the strong increase in LSIL findings among women infected with multiple carcinogenic types could mask and attenuate the increase in the risk of HSIL. Also, we cannot exclude residual confounding by age, since younger age is associated with a higher prevalence of infection with multiple types and a lower prevalence of cervical precancer. Our findings support results of a previous study from the Costa Rica Vaccine Trial that did not find evidence for synergism between types in participants infected with multiple types [10]. Another study from Canada reported increasing risks of HSIL in categories of 1 infecting type, 2–3 infecting types, and 4–6 infecting types, with point estimates of risk higher than what would be expected for an additive effect yielded by infection with multiple carcinogenic genotypes [11]. However, the number of end points in that study was low, and the risk estimates had wide and overlapping CIs. Furthermore, the analysis used women without HPV infection as a reference category, possibly leading to unstable estimates for outcomes (ie, LSIL and HSIL) that are very strongly associated with the presence of HPV.
Our analysis has several strengths. We report the largest sample of cervical cytology specimens with HPV genotyping data and cytology results, which was drawn form a large, well-characterized, registry-based population of women with cervical cytology test results in the state of New Mexico [12]. HPV genotyping was performed using the well-characterized HPV LA in a single, highly experienced HPV laboratory. We also note some limitations of our study. Our study is purely cross-sectional and does not account for prospective risk of disease at this point. Our end points were only based on cytology results. When HSIL is used as a disease end point, the prevalence of cervical precancer is underestimated, because many CIN3 cases are found in women with ASC-US and LSIL [17].
In summary, we demonstrate that infection with multiple carcinogenic types is associated with a higher risk of any cytologic abnormality and with a risk of HSIL but that the risk is lower than what would be expected assuming an additive risk of HSIL for women infected with multiple genotypes. Thus, in the largest study to date with HPV genotyping data and disease end points, we support that there is no evidence for synergy between HPV genotypes during concomitant infection with multiple HPV types.
Supplementary Data
Supplementary materials are available at The Journal of Infectious Diseases online (http://jid.oxfordjournals.org/). Supplementary materials consist of data provided by the author that are published to benefit the reader. The posted materials are not copyedited. The contents of all supplementary data are the sole responsibility of the authors. Questions or messages regarding errors should be addressed to the author.
Notes
Acknowledgments. All authors (N. W., M. N., M. S., L. D., W. C. H., and C. M. W.) reviewed, provided input to, and approved the final manuscript submitted for publication. Members of the New Mexico HPV Pap Registry (NMHPVPR) Steering Committee reviewed and gave input to the manuscript and supported the directions of the NMHPVPR including the evaluations presented in this manuscript. The participating NMHPVPR steering committee members are as follows: Nancy E. Joste, MD, University of New Mexico Health Sciences Center and Tricore Reference Laboratories, Albuquerque, New Mexico; Walter Kinney, MD, Kaiser Permanente Northern California; Cosette M. Wheeler, PhD, University of New Mexico Health Sciences Center; William C. Hunt, MS, University of New Mexico Health Sciences Center; Deborah Thompson, MD, MSPH, New Mexico Department of Health; Susan Baum, MD, MPH, New Mexico Department of Health; Linda Gorgos, MD, MSc, former medical director of the Infectious Disease Bureau, New Mexico Department of Health; Alan Waxman, MD, MPH, University of New Mexico Health Sciences Center; David Espey, MD, Centers for Disease Control and Prevention; Jane McGrath, MD, University of New Mexico Health Sciences Center; Steven Jenison, MD, community member; Mark Schiffman, MD, MPH, National Cancer Institute; Philip Castle, PhD, MPH, Albert Einstein College of Medicine; Vicki Benard, PhD, Centers for Disease Control and Prevention; Debbie Saslow, PhD, American Cancer Society; Jane J. Kim, PhD, Harvard School of Public Health; Mark H. Stoler, MD, University of Virginia; Jack Cuzick, PhD, Wolfson Institute of Preventive Medicine, London; Giovanna Rossi Pressley, MSc, Collective Action Strategies and Robert Wood Johnson Foundation Center for Health Policy at University of New Mexico; and Kevin English, RPh, MPH, Albuquerque Area Southwest Tribal Epidemiology Center. No compensation was received for contributions to this manuscript by any named authors or by members of the NMHPVPR.
Disclaimer. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Financial support. This work was supported by the National Institutes of Health (awards U19AI084081 and R01CA134779 to C. M. W.) and by the Intramural Research Programs of the National Cancer Institute and National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services. HPV Linear Array reagents and equipment to automate HPV genotyping assays were provided by Roche Molecular Systems.
Potential conflicts of interest. C. M. W. has received funds from grants and cooperative agreements from the National Institutes of Health, through the University of New Mexico, related to cervical screening, funds from GSK for HPV vaccine studies, and reimbursements from Roche Molecular Systems for travel related to publication activities and for equipment and reagents for HPV genotyping. All other authors report no potential conflicts.
All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed
References
- 1.Schiffman M, Wentzensen N, Wacholder S, Kinney W, Gage JC, Castle PE. Human papillomavirus testing in the prevention of cervical cancer. J Natl Cancer Inst. 2011;103:368–83. doi: 10.1093/jnci/djq562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Trottier H, Franco EL. The epidemiology of genital human papillomavirus infection. Vaccine. 2006;24(Suppl 1):S1–15. doi: 10.1016/j.vaccine.2005.09.054. [DOI] [PubMed] [Google Scholar]
- 3.Chen HC, Schiffman M, Lin CY, et al. Persistence of type-specific human papillomavirus infection and increased long-term risk of cervical cancer. J Natl Cancer Inst. 2011;103:1387–96. doi: 10.1093/jnci/djr283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kjaer S, Hogdall E, Frederiksen K, et al. The absolute risk of cervical abnormalities in high-risk human papillomavirus-positive, cytologically normal women over a 10-year period. Cancer Res. 2006;66:10630–6. doi: 10.1158/0008-5472.CAN-06-1057. [DOI] [PubMed] [Google Scholar]
- 5.Schiffman M, Glass AG, Wentzensen N, et al. A long-term prospective study of type-specific human papillomavirus infection and risk of cervical neoplasia among 20,000 women in the Portland Kaiser Cohort Study. Cancer Epidemiol Biomarkers Prev. 2011;20:1398–409. doi: 10.1158/1055-9965.EPI-11-0206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Li N, Franceschi S, Howell-Jones R, Snijders PJ, Clifford GM. Human papillomavirus type distribution in 30,848 invasive cervical cancers worldwide: Variation by geographical region, histological type and year of publication. Int J Cancer. 2011;128:927–35. doi: 10.1002/ijc.25396. [DOI] [PubMed] [Google Scholar]
- 7.Wentzensen N, Schiffman M, Dunn T, et al. Multiple human papillomavirus genotype infections in cervical cancer progression in the study to understand cervical cancer early endpoints and determinants. Int J Cancer. 2009;125:2151–8. doi: 10.1002/ijc.24528. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wheeler CM, Hunt WC, Joste NE, Key CR, Quint WG, Castle PE. Human papillomavirus genotype distributions: implications for vaccination and cancer screening in the United States. J Natl Cancer Inst. 2009;101:475–87. doi: 10.1093/jnci/djn510. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wentzensen N, Wilson LE, Wheeler CM, et al. Hierarchical clustering of human papilloma virus genotype patterns in the ASCUS-LSIL triage study. Cancer Res. 2010;70:8578–86. doi: 10.1158/0008-5472.CAN-10-1188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Chaturvedi AK, Katki HA, Hildesheim A, et al. Human papillomavirus infection with multiple types: pattern of coinfection and risk of cervical disease. J Infect Dis. 2011;203:910–20. doi: 10.1093/infdis/jiq139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Trottier H, Mahmud S, Costa MC, et al. Human papillomavirus infections with multiple types and risk of cervical neoplasia. Cancer Epidemiol Biomarkers Prev. 2006;15:1274–80. doi: 10.1158/1055-9965.EPI-06-0129. [DOI] [PubMed] [Google Scholar]
- 12.Wheeler CM, Hunt WC, Cuzick J, et al. A population-based study of human papillomavirus genotype prevalence in the United States: Baseline measures prior to mass human papillomavirus vaccination. Int J Cancer. 2013;132:198–207. doi: 10.1002/ijc.27608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Quint W, Jenkins D, Molijn A, et al. One virus, one lesion–individual components of CIN lesions contain a specific HPV type. J Pathol. 2012;227:62–71. doi: 10.1002/path.3970. [DOI] [PubMed] [Google Scholar]
- 14.van der Marel J, Quint WG, Schiffman M, et al. Molecular mapping of high-grade cervical intraepithelial neoplasia shows etiological dominance of HPV16. Int J Cancer. 2012;131:E946–E953. doi: 10.1002/ijc.27532. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kovacic MB, Castle PE, Herrero R, et al. Relationships of human papillomavirus type, qualitative viral load, and age with cytologic abnormality. Cancer Res. 2006;66:10112–9. doi: 10.1158/0008-5472.CAN-06-1812. [DOI] [PubMed] [Google Scholar]
- 16.Porras C, Rodriguez AC, Hildesheim A, et al. Human papillomavirus types by age in cervical cancer precursors: predominance of human papillomavirus 16 in young women. Cancer Epidemiol Biomarkers Prev. 2009;18:863–5. doi: 10.1158/1055-9965.EPI-08-0951. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Kinney WK, Manos MM, Hurley LB, Ransley JE. Where's the high-grade cervical neoplasia? The importance of minimally abnormal Papanicolaou diagnoses. Obstet Gynecol. 1998;91:973–6. doi: 10.1016/s0029-7844(98)00080-5. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.