Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Jul 1.
Published in final edited form as: J Low Genit Tract Dis. 2010 Jul;14(3):185–195. doi: 10.1097/LGT.0b013e3181cd6d36

Accuracy and Cost-Effectiveness of Cervical Cancer Screening by High-Risk HPV DNA Testing of Self-Collected Vaginal Samples

Akhila Balasubramanian 1,2, Shalini L Kulasingam 3, Atar Baer 1, James P Hughes 4, Evan R Myers 5, Constance Mao 6, Nancy B Kiviat 7, Laura A Koutsky 1
PMCID: PMC2898894  NIHMSID: NIHMS181119  PMID: 20592553

Abstract

Objective

Estimate the accuracy and cost-effectiveness of cervical cancer screening strategies based on high-risk HPV DNA testing of self-collected vaginal samples.

Materials and Methods

A subset of 1,665 women (18-50 years of age) participating in a cervical cancer screening study were screened by liquid-based cytology and by high-risk HPV DNA testing of both self-collected vaginal swab samples and clinician-collected cervical samples. Women with positive/abnormal screening test results and a subset of women with negative screening test results were triaged to colposcopy. Based on individual and combined test results, five screening strategies were defined. Estimates of sensitivity and specificity for cervical intraepithelial neoplasia grade 2 or worse were calculated and a Markov model was used to estimate the incremental cost-effectiveness ratios (ICERs) for each strategy.

Results

Compared to cytology-based screening, high-risk HPV DNA testing of self-collected vaginal samples was more sensitive (68%, 95%CI=58%-78% versus 85%, 95%CI=76%-94%) but less specific (89%, 95%CI=86%-91% versus 73%, 95%CI=67%-79%). A strategy of high-risk HPV DNA testing of self-collected vaginal samples followed by cytology triage of HPV positive women, was comparably sensitive (75%, 95%CI=64%-86%) and specific (88%, 95%CI=85%-92%) to cytology-based screening. In-home self-collection for high-risk HPV DNA detection followed by in-clinic cytology triage had a slightly lower lifetime cost and a slightly higher quality-adjusted life expectancy than did cytology-based screening (ICER of triennial screening compared to no screening was $9,871/QALY and $12,878/QALY, respectively).

Conclusions

Triennial screening by high-risk HPV DNA testing of in-home, self-collected vaginal samples followed by in-clinic cytology triage was cost-effective.

Keywords: cervical cancer, screening, hpv, self-collect, cost-effectiveness

INTRODUCTION

Cytology-based screening programs have led to a dramatic reduction in cervical cancer incidence over the last five decades. Yet, these programs have limitations that have prompted research into alternative screening technologies and strategies. Cytology-based screening is labor- and time-intensive and lacks sensitivity and reproducibility in detecting precancerous cervical lesions or cervical cancer (1-4). Lack of participation in screening is also a challenge. A number of novel approaches to screening are being investigated, including the use of self-collected specimens for high-risk human papillomavirus (HPV) DNA testing.

In the United States, HPV DNA testing is recommended as an adjunct to cytology screening for all women 30 years and older, and reflex testing for HPV DNA is recommended for women with a cytologic finding of atypical squamous cells of undetermined significance (ASC-US) (5). HPV DNA testing has been shown to have higher sensitivity in screening (2, 6-11) and greater reproducibility (12) than cytology testing. It also lends itself to self-testing options. Studies using cervico-vaginal lavage, vaginal swabs or brushes, and tampons (13-29) suggest that there is good agreement between high-risk HPV DNA detection from self-collected vaginal samples vs. clinician-collected cervical samples and high acceptability of vaginal self-collection (20, 26-28). On the other hand, high-risk HPV testing of self-collected vaginal samples, like HPV testing of clinician-collected cervical samples, is less specific than cytology (23, 30), which leads to increased unnecessary diagnostic and treatment costs associated with false-positive test results.

We conducted a screening study in a population of young, sexually active women to determine the accuracy of high-risk HPV DNA testing of self-collected specimens alone or in combination with cytology for cervical cancer screening. We then used the resulting test characteristics as parameters in a mathematical model to estimate the cost-effectiveness of the different screening strategies.

MATERIALS AND METHODS

Study population

This project was part of a previously described (7) larger cervical cancer screening study. 1,665 women were recruited between 2000 and 2002 from among women attending routine gynecologic examinations at Planned Parenthood of Western Washington clinics and by advertisements in local colleges and universities and in newspapers. Women were eligible for participation if they were between 18 and 50 years old, not pregnant, not chronically immunocompromised, and reported no prior treatments for cervical neoplasia. All study procedures were approved by the Human Subjects Division at the University of Washington, and written informed consent was obtained from all research participants.

Study Methods

Clinical procedures for screening, follow-up, treatment, and laboratory methods have been described previously (7). Briefly, women were screened by liquid-based cervical cytology and by high-risk HPV DNA testing (using PCR-based and Hybrid Capture II™ (hc2, Qiagen, Valencia, CA) tests) of cervical samples. For the time period of the study included in this analysis, participants also provided a self-collected vaginal swab sample for HPV DNA testing prior to the clinical examination. Details on self-sample collection and processing are the same as those previously described by our group in a different study (31). Women with a positive/abnormal result on any screening test and a random subset of those with normal screening test results were referred to diagnostic colposcopy and biopsy. At the colposcopy visit, colposcopically-directed biopsy and/or endocervical curettage (ECC) samples were collected from all women; for women with normal colposcopic findings an ectocervical biopsy sample at the 12 o’clock position and/or ECC were collected.

Statistical analysis

The most severe histologic finding from diagnostic and/or treatment visits was used to define disease outcomes. Estimates of sensitivity and specificity for detection of cervical intraepithelial neoplasia grade 2 or more severe disease (≥CIN2) were obtained for five screening strategies that included referral to colposcopy of women with the following test results: (1) a screening cytology showing low-grade squamous intraepithelial lesions (LSIL), atypical glandular cells (AGC) or ≥high-grade SIL (HSIL), or a screening cytology showing ASC-US with positive hc2 test results on the cervical sample (referred to as ‘cytology screening with reflex HPV testing for ASC-US’); (2) a screening cytology showing LSIL, AGC or ≥HSIL, or a screening cytology showing ASC-US with a follow-up cytology showing ASC-US, AGC, LSIL or HSIL (referred to as ‘cytology screening with repeat cytology testing for ASC-US’); (3) positive hc2 test of the self-collected vaginal sample; (4) positive hc2 test of the clinician-collected cervical sample; (5) positive hc2 results on the self-collected vaginal sample and a follow-up cytology showing ASC-US, AGC, LSIL or ≥HSIL (referred to as ‘vaginal HPV DNA screening with cytology triage’).

Sensitivity and specificity estimates were calculated for each screening strategy, and were corrected for verification bias using inverse probability weighting (IPW) as previously described (7). The five ‘screening strata’ used for IPW included women with: (1) negative results on all screening tests; (2) negative result on LBC but positive results for HR HPV DNA in prospectively tested cervical or vaginal sample using hc2 or PCR-based methods; (3) LBC result of ASC-US; (4) LBC result of LSIL; (5) LBC results of AGC or ≥HSIL. HPV DNA and cytology results were the only factors found to be independently associated with both colposcopy attendance and detection of ≥CIN2.

Weighted robust variance estimators were used for calculating 95% confidence intervals (C.I.s) of corrected sensitivity and specificity estimates. A weighted McNemar’s test was used to compare differences between the weighted sensitivity and specificity estimates from different screening strategies. Weights were applied using the pweights command in STATA 9.0 (STATA Corp, College Station, TX), which was used for data analyses.

Mathematical modeling to evaluate cost-effectiveness of screening strategies

A previously described state-transition Markov model (32) was used to estimate the lifetime costs, lifetime risks of cervical cancer, quality-adjusted life expectancies and incremental cost-effectiveness ratios (ICER) of the screening strategies described above, using a societal perspective. The model simulates a cohort of women beginning at age 12 years and followed through age 85 years. Each year women are at risk of being infected with HPV. Women who are infected can either progress to CIN (1, 2-3) or have their infection clear. Women with CIN can either clear their disease or have their disease progress to a more severe disease state (CIN 2-3 or cancer). Women with cancer can have their disease detected during screening or, in the absence of screening, through presentation with symptoms. Women with cancer can die of cancer based on 5-year survival probabilities. Each year women are also at risk of dying of other causes or undergoing hysterectomy for benign reasons. Stage-specific survival estimates for cervical cancer (33) and age-specific probability of death due to causes other than cervical cancer (34,35) were taken from the SEER registry. Age-specific rates of benign hysterectomy were taken from the published literature (36).

Assumptions about compliance and clinical management of disease

Women were assumed to be screened between the ages of 18 and 85 years, with one of the five previously described strategies conducted at intervals of every 1, 2 or 3 years. It was assumed that cervical screening test samples would be collected by a clinician while vaginal test samples would be self-collected by women in their homes. The compliance rate with screening was assumed to be 80% in our simulated population, based on reported screening rates in the U.S.(37, 38). Women with abnormal screening test results were referred for colposcopy and biopsy or assumed to undergo another screening test prior to referral to colposcopy. Compliance with colposcopy and treatment was assumed to be 100% in keeping with the literature: this allows for the assumption that screening strategies are being compared while holding constant external factors not related to the screening test performance. Disease status was assumed to be histologically verified among all women attending the colposcopy visit. Due to the variable management guidelines for women with CIN 1, which recommend treatment only in a subset of cases (39), these women were assumed not to undergo treatment but to return to regular screening. All women with a cytology result of HSIL or histology results ≥CIN2 were assumed to receive treatment (Loop Electrosurgical Excision Procedure [LEEP] for HSIL cytology, cone biopsies or LEEP for CIN2 and CIN3 and stage appropriate treatment for cancer). Women with cancer were assumed to receive stage-specific treatment.

Costs and Utilities

Base case assumptions and ranges for relevant cost parameters are presented in table 1. Direct costs included costs for medical procedures and clinicians’ time. Indirect costs included the cost of time spent in clinic and in transportation for clinic-based procedures, and time spent on self-collection and mailing of specimens for home-based procedures.

Table 1.

Estimated Costs of Screening, Diagnostic, and Treatment Procedures and Estimated Compliance With Screening and Diagnostic Follow-up

Procedure Base case estimatea Plausible rangea
Screening costs
 Liquid-based cytologyb 28 14–56c
 Pathologist review for abnormal—liquid-based cytologyb 17 8–34c
 HPV DNA test (hc2)b 49 24–98c
 Office visitb 63 32–126c
 Patient’s time for screening clinic visitd 18 for women <25 y 12–54c
28 for women ≥25 y
 Patient’s time for self-collection at homee 4 for women <25 y 2–14c
7 for women ≥25 y
 Shipping hc2 kit between patient’s home and clinice 20.00 10–40c
Diagnostic costs
 Colposcopy and biopsyb 237 118–474c
 Patient’s time for diagnostic visitf 24 for women <25 y; 12–74c
37 for women ≥25 y
Treatment costs
 LEEP treatmentb 410 205–820c
 Patient’s time for LEEP treatment visitf 24 for women <25 y; 12–74c
37 for women ≥25 y
 Treatment and follow-up for ≥CIN 2b,g 3,501 1,750–7,002c
 Treatment for stage I cervical cancerb,g 26,612 13,306–53,224c
 Treatment for stage II cervical cancerb,g 28,482 14,241–56,964c
 Treatment for stage III cervical cancerb,g 33,451 16,725–66,902c
 Treatment for stage IV cervical cancerb,g 45,618 22,809–91,236c
 Terminal careb 23,050 11,525–46,100c
Compliance with screening 80.00% 30.00%–100.00%h
Compliance with colposcopy
 ASCUS Pap at screening 100% 70%–100%i
 LSIL Pap at screening 100% 70%–100%i
 HSIL Pap at screening 100% 70%–100%i
 Cancer Pap at screening 100% 70%–100%i
 High-risk HPV DNA detected at screening 100% 70%–100%i
Open in a new tab
a

All costs were inflated to 2007 US $ using the medical component of the consumer price index from the Bureau of Labor Statistics [42].

b

Direct medical costs for screening [43-45], diagnosis [46], LEEP treatment [45], cancer treatment [46], and terminal care [4] were estimated from the clinical laboratory and physician fee schedules of Medicare [43, 44] and from the published literature [4, 45-47].

c

All costs were univariately varied from 50% to 200% of their base case values.

d

The estimated time spent undergoing clinic-based screening, including time spent in transportation and waiting in the clinic, was taken from the published literature [48]. The cost of that time was estimated by applying the national age-specific median wage rate for women [49] to the estimated time associated with the visit.

e

The estimated time spent undergoing home-based screening was estimated based on a study performed by our group in which participants self-collected vaginal HPV specimens at home and mailed them to the clinic [31]. On the basis of this study, it was estimated to take 20 minutes for participants to complete the self-collection and mail the HPV DNA test. The cost of this time was computed by applying the median national age-adjusted wage rate for women [49]. The estimated cost of shipping the self-collected HPV DNA specimen to the clinic was also based on our group’s experience in the previous study [31].

f

The time spent undergoing colposcopy-biopsy and LEEP treatment procedures was estimated based on our experience with the time spent in the clinic for these procedures (a conservative estimate of 1 h) and our estimate of time spent in transportation associated with these procedures. As the published transportation time relates to cervical cancer screening rather than diagnosis or treatment, and because fewer clinics in the community offer diagnostic and treatment services, we assumed a longer transportation time here (40 min versus the 18.8 min reported [48] for one-way transportation to a screening visit). The cost of time was computed by applying the national age-specific median wage rate for women [49] to the estimated time.

g

The published values of costs of treatment for treatment procedures other than LEEP already include indirect time costs [46]. For the LEEP treatment, we added our estimate of indirect time cost for LEEP treatment visit to the published cost of LEEP treatment [45].

h

The rate of compliance with screening was varied between 30% and 100% based on a report that 31.2% enrolled in a US health plan attended cervical cancer screening in 1 year [50].

i

For compliance with colposcopy, the base case value of 100% was chosen in keeping with convention in the literature [46, 51]: this allows for a comparison between different screening strategies while maintaining constant those factors not directly related to the screening test. In sensitivity analyses, rates of compliance with colposcopy were varied between 70% and 100%. In our screening study, the observed rates of compliance with colposcopy were related to the screening test result as follows: 75% among women with ASCUS Pap and positive HPV test results, 79% among women with LSIL, 91% among women with HSIL, and 100% among women with cancer Pap test results. Among women with positive HPV test results, 69% attended colposcopy. On the basis of this observation of differential compliance among women with different levels of abnormal screening results, we performed a sensitivity analysis using observed compliance rates because differential follow-up of screening test results could influence the relative cost-effectiveness of screening strategies.

Estimates of utilities and disutilities for calculation of QALYs were based on a time trade-off study of women’s perceptions of health states associated with cervical cancer (40). Utilities associated with cancer were 0.76 for stage I and 0.67 for stages (II-IV); these utilities were applied for 5 years. A disutility of 0.02, which is associated with undergoing cervical cancer screening (40), was applied towards false-positive screening test results. This disutility was applied for a period of 54 days, the median time interval between screening and colposcopy visits for women with false-positive screening test results in our study. All costs and outcomes were discounted at 3% (41).

Sensitivity analyses

Sensitivity analyses were performed to determine the impact of univariately varying sensitivity and specificity estimates, compliance with screening, compliance with follow-up, and individual costs (see table 1 for ranges of values evaluated). The estimates of sensitivity and specificity were simultaneously varied between the bounds of their 95% confidence intervals under the assumption that gains in sensitivity are at the expense of specificity and vice versa.

Additional analyses included: (i) cost of clinician time spent counseling HPV positive women (estimated to be twenty minutes), (ii) disutility associated with lack of clinician contact in home-based tests (a disutility of 0.02, which is associated with undergoing screening, was applied as an estimate for a one-month period following screening), and (iii) cost of a follow-up cytology test for women with HPV-positive, cytology-negative results for the vaginal HPV screening with cytology triage strategy. (As this strategy is not clinically recommended and no relevant guidelines exist, we considered a follow-up cytology test with the assumption that women with HPV-positive, cytology-negative results might require some type of follow-up. In the base-case model, such women are assumed to return to regular screening.)

RESULTS

Characteristics of study population

The 1,665 study participants were predominantly non-Hispanic (95.8%) and White (73.8%), with a median age of 23 years. They had a lifetime median of 6 male sex partners and 36.8% reported having had 10 or more sex partners. Cytology showed ASC-US or greater in 21.8%, and high-risk HPV DNA was detected by hc2 in 28.4% of cervical and 33.1% of vaginal samples at screening. 504 women (30.3%) met the referral criteria and underwent colpo-biopsy. 100 women (Figure 1) were found to have histologically-confirmed ≥CIN2, indicating that the prevalence of ≥CIN2 was high (7.8%) in this young, sexually experienced population.

Figure 1. Attendance at colposcopy-biopsy and detection of histologically-confirmed CIN2 and ≥CIN3 among women stratified by screening test results.

Figure 1

Open in a new tab

aISCC = invasive squamous cell carcinoma; HSIL = high grade squamous intraepithelial lesions; AGUS = atypical glandular cells of unknown significance; LSIL = low grade squamous intraepithelial lesions; ASC-US = atypical squamous cells of unknown significance; CIN = cervical intraepithelial neoplasia grade 2; ≥CIN3 = cervical intraepithelial neoplasia grade 3 or higher

Sensitivity and specificity of screening strategies

The sensitivity of self-collected vaginal HPV DNA screening (84.8%) was significantly greater than that of the cytology-based strategies (68.1%) and similar to that of cervical HPV DNA screening (93.8%). Vaginal and cervical HPV DNA screening had similar specificities (72.9% to 73.6%), which were significantly lower than those of the cytology-based strategies (88.6% to 90.6%). The sensitivity (75.2%) and specificity (88.3%) of vaginal HPV DNA screening with cytology triage was comparable to those of cytology-based strategies (Table 2).

Table 2.

Performance of Different Cytology- and HPV-Based Screening Strategies for Detection of Histologically Confirmed ≥CIN 2

Screening strategy (criterion for referral to colposcopy) Weighted sensitivity (95% CI) p for sensitivity (vs strategy 1, strategy 2)a Weighted specificity (95% CI) p for specificity (vs strategy 1, strategy 2)a Referred for colposcopy, % Negative predictive value (95% CI) Positive predictive value (95% CI)
1. (a) Liquid-based cytology result of ≥LSIL or (b) Liquid-based cytology result of ASCUS and positive HPV DNA result from hc2 testing of cervical sample 68.1 (57.9–78.4) 1.00b 88.6 (86.0–91.3) .001b 16.6 97.1 (95.8–98.3) 33.6 (26.9–40.2)
2. (a) Liquid-based cytology result of ≥LSIL or (b) Liquid-based cytology result of ASCUS and a repeat liquid-based cytology result of ≥ASCUS 68.1 (57.9–78.4) 1.00b 90.6 (88.3–93.0) .001b 13.8 97.1 (95.9–98.4) 37.9 (30.6–45.3)
3. Positive HPV DNA result from hc2 testing of self-collected vaginal sample 84.8 (75.5–94.1) .027, .058 72.9 (66.6–79.2) <.0001, <.0001 33.1 98.3 (97.1–99.5) 20.4 (15.18–25.6)
4. Positive HPV DNA result from hc2 testing of clinician-collected cervical sample 93.8 (88.3–99.3) .003, .001 73.6 (67.9–79.4) <.0001, <.0001 31.8 99.3 (98.7–100.0) 22.8 (17.7–27.9)
5. (a) Positive HPV DNA result from hc2 testing of self-collected vaginal sample and (b) Liquid-based cytology result of ≥ASCUS 75.2 (64.3–86.1) .292, .442 88.3 (84.7–92.0) .212, .548 18.6 97.8 (96.5–99.0) 34.5 (25.7–43.3)
Open in a new tab
a

The 2 p values refer to the result of McNemar test comparing the sensitivity or specificity associated with the given screening strategy to the corresponding test parameter for (1) strategy 1 and (2) strategy 2. Although the sensitivity estimate is identical for strategies 1 and 2, in comparison with other strategies, the p values associated with strategies 1 and 2 are not identical: this occurred because the various subsets of data pertaining to the different comparisons contain varying levels of missing data points.

b

p value compares the sensitivity or specificity associated with strategy 1 versus the corresponding test parameter for strategy 2.

Cost-effectiveness of screening strategies

In-home vaginal HPV DNA screening with in-clinic cytology triage was the least costly strategy and also more effective in terms of QALYs than all other strategies. The only exception to this was in the case of the triennial screening interval, where in-clinic cervical HPV DNA screening was associated with the highest effectiveness. Cervical HPV DNA screening was, however, also the most costly strategy: compared to triennial vaginal HPV DNA screening with cytology triage, triennial cervical HPV DNA screening had an ICER of $238,706/QALY (Figure 2). Compared to no screening, triennial vaginal HPV DNA screening with cytology triage was associated with a 75% reduction in lifetime risk of cervical cancer (3.61% vs. 0.91%) and an ICER of $9,871/QALY. Biennial screening with this strategy had an ICER of $70,151/QALY compared to triennial screening. Annual screening was very costly for this and all other strategies, with ICERs exceeding $300,000/QALY.

Figure 2. Cost-effectiveness of different screening strategies: results of base case analysis.

Figure 2

Open in a new tab

Note: Collection of vaginal samples is assumed to be performed by women in their homes; collection of cytology samples is assumed to be performed in a clinic setting

If compared directly to no screening, then both cytology-based strategies were cost-effective, with the reflex HPV DNA triage option being the least costly. Triennial and biennial screening with this strategy had ICERs of $12,878/QALY and $18,051/QALY, respectively, compared to no screening. Compared to biennial cytology screening with HPV DNA triage for ASC-US, biennial vaginal HPV DNA screening with cytology triage was associated with a gain in quality-adjusted life expectancy of 1.10 days and a reduction in lifetime costs of $199.

Findings from sensitivity analyses

1) Effect of compliance with follow-up

Our findings were most sensitive to assumptions about compliance with follow-up. If compliance with colposcopy was progressively lowered from the base-case value of 100% to 80%, triennial and biennial stand-alone in-home vaginal HPV screening without cytology triage were identified as potentially cost-effective, with an ICER of $10,508/QALY for triennial screening compared to no screening. When we applied the compliance rates for colposcopy observed in our screening study (specified in Table 1), which varied between women with different types of screening test results, vaginal HPV-based strategies with or without cytology triage remained cost-effective, and cytology screening with repeat cytology testing for ASC-US also became cost-effective with ICERs ranging from $78,770/QALY to $187,540/QALY depending on the screening interval.

2) Other influential factors in sensitivity analyses

Altering assumptions regarding the administration of HPV- and home-based strategies affected our findings in the following ways: (1) if it was assumed that women in the vaginal HPV screening with cytology triage strategy would undergo repeat cytology testing for HPV-positive, cytology-negative screening test results, then triennial stand-alone vaginal HPV screening became the least costly strategy but triennial and biennial vaginal HPV with cytology triage also remained cost-effective, with ICERs of $19,127/QALY and $75,839/QALY, respectively; (2) if a disutility associated with lack of clinician contact was included in home-based strategies then biennial vaginal HPV screening with cytology triage became less effective and more costly than triennial in-clinic Pap- or HPV-based strategies; (3) including a cost for clinician time for counseling HPV-positive women did not appreciably alter the results.

Acceptability of Self-Collected Vaginal Sampling

Of the 1,665 eligible women, 249 (14.9%) did not provide a self-collected vaginal sample, which most often occurred because of clinical time constraints. Women who did and did not provide a self-collected sample had similar demographic and clinical characteristics. Four hundred and four participants of the 504 who attended colposcopy were interviewed regarding their experience with self-collecting the vaginal sample during their previous screening visit and 246 (61%) responded that if given the option of a safe and accurate home-based self-test, they would prefer it over clinic-based testing, the reasons being greater ease and convenience; avoidance of the inconveniences of transportation and time required for a clinic visit; greater privacy of the home-based self-test; and fear or embarrassment associated with doctors and gynecologic exams. Of the 158 women who preferred a clinic-based test, 98 (62%) felt that it would be more accurate. Twelve women reported that they would want to benefit from having questions answered by the clinician or of undergoing a more thorough examination in which other, unrelated health problems might potentially be diagnosed.

DISCUSSION

In this population of young, sexually active women in the U.S., a combined strategy of vaginal high-risk HPV DNA screening with cytology triage of HPV positive women had comparable sensitivity and specificity to cytology-based screening. Assuming that vaginal self-sampling would be home-based and cytology testing clinic-based, this combined strategy was potentially cost-effective when performed triennially or biennially. Compared to absence of screening, triennial screening of home-based, self-collected vaginal samples for HPV DNA with in-clinic cytology triage had an ICER of $9,871/QALY. Triennial in-clinic cytology screening with reflex HPV DNA testing for ASC-US had an ICER of $12,878/QALY compared to no screening. Home-based self-collected HPV screening may be considered as a cost-effective screening alternative to increase convenience and accessibility of existing cytology-based screening programs.

To our knowledge, there have been no published studies of HPV screening of self-collected vaginal samples in a screening population in the United States. Investigations in China (23) and South Africa (30) indicated that this strategy performed favorably as an alternative to cytology-based screening. These studies were conducted among populations that were markedly older than ours (median ages of 42 and 39 years, respectively, versus 23 years in our study) and, given screening practices in these countries, less frequently screened. The estimated sensitivity of cytology screening in our study was similar to estimates reported for other well-screened populations in North America and Europe (6). In keeping with our findings, Belinson et al and Wright et al reported that vaginal HPV DNA screening was less specific than cytology screening. Neither of these studies investigated a combined strategy of vaginal HPV DNA testing with cytology triage and referral to colposcopy on the basis of positive/abnormal results from both these tests. In our study, this strategy showed similar sensitivity and specificity to cytology-based strategies.

Cost-effectiveness studies of HPV-based cervical cancer screening in industrialized countries have generally focused on incorporating HPV testing into existing cytology-based screening programs. One U.S study of primary screening with HPV showed that, compared to cytology-based screening, HPV screening of clinician-collected cervical samples was associated with a greater reduction in lifetime risk of cervical cancer but was not cost-effective (51). This observation was replicated in our study. When HPV screening is clinic-based, its cost is comparable to that of cytology screening. Furthermore, the relatively low specificity of HPV screening leads to a high cost of unnecessary diagnostic tests. By using in-home, self-collected vaginal samples for HPV screening, the large reduction in lifetime risk of cervical cancer is maintained by the high sensitivity of the strategy, and the cost of screening is reduced substantially by eliminating the need for an office visit. The specificity of vaginal HPV DNA-based screening was improved to the level of cytology screening by using cytology testing to triage HPV-positive women to colposcopy. This improved specificity further reduced the costs associated with this screening strategy. This combination of relatively low costs and high sensitivity and specificity made this strategy less costly and more effective than all others investigated.

There were limitations to this study. First, our study participants performed the vaginal self-collection in the clinic rather than at home. Our conclusions regarding this test in a home-based setting are based on an assumption that it would perform similarly in both settings. In a previous study by our group, the prevalence of HPV DNA was statistically similar between vaginal samples self-collected at home vs. in the clinic (31). Other investigators have reported high specimen adequacy as indicated by the level of β-globin in vaginal samples self-collected at home and transported to the clinic (52, 53), as well as increased HPV detection in these samples compared to clinician-collected cervical samples (53). Second, our model did not take into account the potential impacts of HPV vaccination of young women on the design, implementation and performance of screening. It is beyond the scope of this article to discuss the effect of vaccination on screening performance, and readers are referred to two detailed analyses in the literature (54, 55). Briefly, the positive predictive values of both cytology and HPV screening are likely to decrease in the era of HPV vaccination. It is more difficult to predict whether and how the sensitivities and specificities of screening tests would change, but it seems reasonable that the direction of change would be similar for cytology and HPV screening. More importantly, researchers predict that in the era of HPV vaccines, HPV testing will take on an increasingly important role in primary screening as well as in disease surveillance and monitoring of vaccine effectiveness (54, 55). Under these circumstances, a self-test for HPV could be advantageous. It is likely that once vaccine coverage is widespread, the interval of screening will be extended and screening might begin at an older age (54, 56). Third, it is possible that some disease may have remained undetected due to the fact that histology-confirmed disease outcome was not ascertained in all women, and only a single biopsy was used to confirm disease status in women with normal colposcopic findings. Incomplete ascertainment of disease outcome could bias sensitivity estimates towards being artificially inflated for all screening tests. Also, our cost-effectiveness model applied the same estimates of sensitivity and specificity across all age groups. Our sensitivity analyses showed, however, that vaginal HPV DNA screening at home with cytology triage was consistently cost-effective when estimates of sensitivity and specificity were varied. A higher estimate of specificity for the vaginal HPV DNA test, which would be expected in older age groups, caused home-based vaginal HPV DNA screening to become cost-effective even without cytology triage. Fourth, our model did not account for the possibility that home-based HPV screening with cytology triage could lead to fewer interactions with health care providers, and thus might be associated with increased costs due to missed opportunities to diagnose and manage unrelated medical conditions. On the other hand, clinic visits are also associated with an unknown risk of receiving unnecessary and potentially harmful interventions (e.g., HRT). Understanding the impact of these difficult-to-quantify indirect costs of gynecological visits will become more important as clinical recommendations for age of initiation and intervals of screening are increased. Finally, the results of our study may not be generalizable to other populations of women who are older, have a lower risk of HPV infection, or have never been screened..

The field of sexually transmitted diseases is increasingly moving towards the use of self-collected vaginal swab samples in screening and surveillance. The findings of this study suggest that cervical cancer screening strategies that are based on self-collection of vaginal swab samples at home could perform with comparable, and possibly even greater, effectiveness and cost-effectiveness compared to existing clinic-based cytology programs. We found that participants were receptive to the idea of a self-test if this test were safe and reliable. Other investigators have reported that self-sampling for HPV is not only acceptable but often preferred by women compared to clinician-sampling (20, 21, 26, 28, 57, 58). In a recent workshop held at the National Institutes of Health, a panel of experts recommended the use of self-collected vaginal specimens for detecting chlamydial and gonorrheal infections (59). They further emphasized a more widespread use of self-sampling to detect other genital pathogens. The FDA has already approved self-sampling kits for HIV and Hepatitis C testing (59). The natural next step would be to develop a common sampling device and collection medium that could aid in integrating testing for various genital infections including HPV. With the advent of the HPV vaccine, many changes are anticipated in the protocol and schedule for cervical cancer screening. This is an opportune time to determine the feasibility and impact of incorporating self-testing for high-risk HPV DNA and for other genital infections into our public health programs. Studies to evaluate effectiveness of HPV self-screening in other populations and assess women’s and clinicians’ preferences, as well as larger field trials and feasibility assessments are needed as the next steps before translating the findings of our study into public health policy and programming.

Acknowledgments

The authors would like to acknowledge Amy Bonney, Laura Merrell, Kimberle Tomlinson, and Nicolas Seal for their contributions towards patient recruitment and clinical and logistical support. We also acknowledge Qinghua Feng, Donna Kenney, Jane Kuypers, and the staff at the HPV virology laboratory at Harborview Medical Center, Seattle, WA.

Source of Financial Support: This study was funded through the National Cancer Institute, grant number CA34493

References

  • 1.Stoler MH, Schiffman M. Interobserver reproducibility of cervical cytologic and histologic interpretations: realistic estimates from the ASCUS-LSIL Triage Study. J Natl Cancer Inst. 2001;285:1500–5. doi: 10.1001/jama.285.11.1500. [DOI] [PubMed] [Google Scholar]
  • 2.Cuzick J. Screening for cervical cancer. In: Rohan TE, Shah KV, editors. Cervical Cancer: From Etiology to Prevention. Norwell: Kluwer Academic Publishers; 2004. pp. 261–93. [Google Scholar]
  • 3.Nanda K, McCrory DC, Myers ER, Bastian LA, Hasselblad V, Hickey JD, et al. Accuracy of the Papanicolaou test in screening for and follow-up of cervical cytologic abnormalities: a systematic review. Ann Intern Med. 2000;132:810–19. doi: 10.7326/0003-4819-132-10-200005160-00009. [DOI] [PubMed] [Google Scholar]
  • 4.McCrory DC, Matchar DB, Bastian L, Datta S, Hasselblad V, Hickey J, et al. Evaluation of cervical cytology: Evidence report/technology assessment no. 5. Rockville, MD: Agency for Health Care Policy and Research; 1999. AHCPR Publication No. 99-E010. [PMC free article] [PubMed] [Google Scholar]
  • 5.Wright TC, Jr, Massad LS, Dunton CJ, Spitzer M, Wilkinson EJ, Solomon D. 2006 consensus guidelines for the management of women with abnormal cervical cancer screening tests. Am J Obstet Gynecol. 2007;197:346–55. doi: 10.1016/j.ajog.2007.07.047. [DOI] [PubMed] [Google Scholar]
  • 6.Cuzick J, Clavel C, Petry KU, Meijer CJ, Hoyer H, Ratnam S, et al. Overview of the European and North American studies on HPV testing in primary cervical cancer screening. Int J Cancer. 2006;119:1095–101. doi: 10.1002/ijc.21955. [DOI] [PubMed] [Google Scholar]
  • 7.Kulasingam SL, Hughes JP, Kiviat NB, Mao C, Weiss NS, Kuypers JM, et al. Evaluation of human papillomavirus testing in primary screening for cervical abnormalities: comparison of sensitivity, specificity, and frequency of referral. J Natl Cancer Inst. 2002;288:1749–57. doi: 10.1001/jama.288.14.1749. [DOI] [PubMed] [Google Scholar]
  • 8.Ratnam S, Franco EL, Ferenczy A. Human papillomavirus testing for primary screening of cervical cancer precursors. Cancer Epidemiol Biomarkers Prev. 2000;9:945–51. [PubMed] [Google Scholar]
  • 9.Clavel C, Masure M, Bory JP, Putaud I, Mangeonjean C, Lorenzato M, et al. Human papillomavirus testing in primary screening for the detection of high-grade cervical lesions: a study of 7932 women. Br J Cancer. 2001;84:1616–23. doi: 10.1054/bjoc.2001.1845. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Cuzick J, Beverley E, Ho L, Terry G, Sapper H, Mielzynska I, et al. HPV testing in primary screening of older women. Br J Cancer. 1999;81:554–8. doi: 10.1038/sj.bjc.6690730. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Schiffman M, Herrero R, Hildesheim A, Sherman ME, Bratti M, Wacholder S, et al. HPV DNA testing in cervical cancer screening: results from women in a high-risk province of Costa Rica. J Natl Cancer Inst. 2000;283:87–93. doi: 10.1001/jama.283.1.87. [DOI] [PubMed] [Google Scholar]
  • 12.Castle PE, Wheeler CM, Solomon D, Schiffman M, Peyton CL. Interlaboratory reliability of Hybrid Capture 2. Am J Clin Pathol. 2004;122:238–45. doi: 10.1309/BA43-HMCA-J26V-WQH3. [DOI] [PubMed] [Google Scholar]
  • 13.Vermund SH, Schiffman MH, Goldberg GL, Ritter DB, Weltman A, Burk RD. Molecular diagnosis of genital human papillomavirus infection: comparison of two methods used to collect exfoliated cervical cells. Am J Obstet Gynecol. 1989;160:304–8. doi: 10.1016/0002-9378(89)90430-4. [DOI] [PubMed] [Google Scholar]
  • 14.Morrison EA, Goldberg GL, Hagan RJ, Kadish AS, Burk RD. Self-administered home cervicovaginal lavage: a novel tool for the clinical-epidemiologic investigation of genital human papillomavirus infections. Am J Obstet Gynecol. 1992;167:104–7. doi: 10.1016/s0002-9378(11)91637-8. [DOI] [PubMed] [Google Scholar]
  • 15.Goldberg GL, Vermund SH, Schiffman MH, Ritter DB, Spitzer C, Burk RD. Comparison of cytobrush and cervicovaginal lavage sampling methods for the detection of genital human papillomavirus. Am J Obstet Gynecol. 1989;161:1669–72. doi: 10.1016/0002-9378(89)90947-2. [DOI] [PubMed] [Google Scholar]
  • 16.Coutlee F, Hankins C, Lapointe N. Comparison between vaginal tampon and cervicovaginal lavage specimen collection for detection of human papillomavirus DNA by the polymerase chain reaction. The Canadian Women’s HIV Study Group. J Med Virol. 1997;51:42–7. doi: 10.1002/(sici)1096-9071(199701)51:1<42::aid-jmv7>3.0.co;2-s. [DOI] [PubMed] [Google Scholar]
  • 17.Fairley CK, Chen S, Tabrizi SN, Quinn MA, McNeil JJ, Garland SM. Tampons: a novel patient-administered method for the assessment of genital human papillomavirus infection. J Infect Dis. 1992;165:1103–6. doi: 10.1093/infdis/165.6.1103. [DOI] [PubMed] [Google Scholar]
  • 18.Moscicki AB. Comparison between methods for human papillomavirus DNA testing: a model for self-testing in young women. J Infect Dis. 1993;167:723–5. doi: 10.1093/infdis/167.3.723. [DOI] [PubMed] [Google Scholar]
  • 19.Gravitt PE, Lacey JV, Jr, Brinton LA, Barnes WA, Kornegay JR, Greenberg MD, et al. Evaluation of self-collected cervicovaginal cell samples for human papillomavirus testing by polymerase chain reaction. Cancer Epidemiol Biomarkers Prev. 2001;10:95–100. [PubMed] [Google Scholar]
  • 20.Hillemanns P, Kimmig R, Huttemann U, Dannecker C, Thaler CJ. Screening for cervical neoplasia by self-assessment for human papillomavirus DNA. Lancet. 1999;354:1970. doi: 10.1016/s0140-6736(99)04110-0. [DOI] [PubMed] [Google Scholar]
  • 21.Dannecker C, Siebert U, Thaler CJ, Kiermeir D, Hepp H, Hillemanns P. Primary cervical cancer screening by self-sampling of human papillomavirus DNA in internal medicine outpatient clinics. Ann Oncol. 2004;15:863–9. doi: 10.1093/annonc/mdh240. [DOI] [PubMed] [Google Scholar]
  • 22.Belinson J, Qiao YL, Pretorius R, Zhang WH, Elson P, Li L, et al. Shanxi Province Cervical Cancer Screening Study: A cross-sectional comparative trial of multiple techniques to detect cervical neoplasia. Gynecol Oncol. 2001;83:439–44. doi: 10.1006/gyno.2001.6370. [DOI] [PubMed] [Google Scholar]
  • 23.Belinson JL, Qiao YL, Pretorius RG, Zhang WH, Rong SD, Huang MN, et al. Shanxi Province cervical cancer screening study II: Self-sampling for high-risk human papillomavirus compared to direct sampling for human papillomavirus and liquid based cervical cytology. Int J Gynecol Cancer. 2003;13:819–26. doi: 10.1111/j.1525-1438.2003.13611.x. [DOI] [PubMed] [Google Scholar]
  • 24.Lorenzato FR, Singer A, Ho L, Santos LC, Batista Rde L, Lubambo TM, et al. Human papillomavirus detection for cervical cancer prevention with polymerase chain reaction in self-collected samples. Am J Obstet Gynecol. 2002;186:962–8. doi: 10.1067/mob.2002.122390. [DOI] [PubMed] [Google Scholar]
  • 25.Kahn JA, Slap GB, Huang B, Rosenthal SL, Wanchick AM, Kollar LM, et al. Comparison of adolescent and young adult self-collected and clinician-collected samples for human papillomavirus. Obstet Gynecol. 2004;103:952–9. doi: 10.1097/01.AOG.0000124569.61462.8d. [DOI] [PubMed] [Google Scholar]
  • 26.Nobbenhuis MA, Helmerhorst TJ, van den Brule AJ, Rozendaal L, Jaspars LH, Voorhorst FJ, et al. Primary screening for high risk HPV by home obtained cervicovaginal lavage is an alternative screening tool for unscreened women. J Clin Pathol. 2002;55:435–9. doi: 10.1136/jcp.55.6.435. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Harper DM, Noll WW, Belloni DR, Cole BF. Randomized clinical trial of PCR-determined human papillomavirus detection methods: self-sampling versus clinician-directed--biologic concordance and women’s preferences. Am J Obstet Gynecol. 2002;186:365–73. doi: 10.1067/mob.2002.121076. [DOI] [PubMed] [Google Scholar]
  • 28.Sellors JW, Lorincz AT, Mahony JB, Mielzynska I, Lytwyn A, Roth P, et al. Comparison of self-collected vaginal, vulvar and urine samples with physician-collected cervical samples for human papillomavirus testing to detect high-grade squamous intraepithelial lesions. CMAJ. 2000;163:513–8. [PMC free article] [PubMed] [Google Scholar]
  • 29.Agorastos T, Dinas K, Lloveras B, Font R, Kornegay JR, Bontis J, et al. Self-sampling versus physician-sampling for human papillomavirus testing. Int J STD AIDS. 2005;16:727–9. doi: 10.1258/095646205774763225. [DOI] [PubMed] [Google Scholar]
  • 30.Wright TC, Jr, Denny L, Kuhn L, Pollack A, Lorincz A. HPV DNA testing of self-collected vaginal samples compared with cytologic screening to detect cervical cancer. J Natl Cancer Inst. 2000;283:81–6. doi: 10.1001/jama.283.1.81. [DOI] [PubMed] [Google Scholar]
  • 31.Winer RL, Feng Q, Hughes JP, Yu M, Kiviat NB, O’Reilly S, et al. Concordance of self-collected and clinician-collected swab samples for detecting human papillomavirus DNA in women 18 to 32 years of age. Sex Transm Dis. 2007;34:371–7. doi: 10.1097/01.olq.0000240315.19652.59. [DOI] [PubMed] [Google Scholar]
  • 32.Myers ER, McCrory DC, Nanda K, Bastian L, Matchar DB. Mathematical model for the natural history of human papillomavirus infection and cervical carcinogenesis. Am J Epidemiol. 2000;151:1158–71. doi: 10.1093/oxfordjournals.aje.a010166. [DOI] [PubMed] [Google Scholar]
  • 33.Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov) SEER*Stat Database: Incidence - SEER 17 Regs Limited-Use, 2006 sub (1973-2004 varying), National Cancer Institute, DCCPS, Surveillance Research Program, Cancer Statistics Branch, released April 2007, based on the November 2006 submission.
  • 34.Surveillance, Epidemiology, and End Results (SEER) Program (www.seer.cancer.gov) SEER*Stat Database: Mortality - All COD, Public-Use With State, Total U.S. (1969-2004), National Cancer Institute, DCCPS, Surveillance Research Program, Cancer Statistics Branch, released April 2007. Underlying mortality data provided by NCHS (www.cdc.gov/nchs).
  • 35.http://seer.cancer.gov/cgi-bin/csr/1975_2004/search.pl#results.
  • 36.Elbasha EH, Dasbach EJ, Insinga RP. Model for assessing human papillomavirus vaccination strategies. Emerg Infect Dis. 2007;13:28–41. doi: 10.3201/eid1301.060438. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Blackman DK, B E, Miller DS. Trends in self-reported use of mammograms (1989-1997) and Papanicolaous tests (1991-1997): Behavioral Risk Factor Surveillance System. Mob Motral Wkly Rep CDC Surveill Summ. 1999;48:1–22. [PubMed] [Google Scholar]
  • 38.Solomon D, Breen N, McNeel T. Cervical cancer screening rates in the United States and the potential impact of implementation of screening guidelines. CA Cancer J Clin. 2007;57:105–11. doi: 10.3322/canjclin.57.2.105. [DOI] [PubMed] [Google Scholar]
  • 39.Wright TC, Jr, Massad LS, Dunton CJ, Spitzer M, Wilkinson EJ, Solomon D. 2006 consensus guidelines for the management of women with cervical intraepithelial neoplasia or adenocarcinoma in situ. Am J Obstet Gynecol. 2007;197:340–5. doi: 10.1016/j.ajog.2007.07.050. [DOI] [PubMed] [Google Scholar]
  • 40.Myers ER, Green S, Lipkus I. Patient preferences for health states related to HPV infection. Visual analog scales versus time trade-off elicitation. Paper presented at: 21st International Papillomavirus Conference; Mexico City. 2004. [Google Scholar]
  • 41.Gold MR, Siegel JL, Russell LB, Weinstein MC, editors. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press Inc; 1996. [Google Scholar]
  • 42.http://www.bls.gov/cpi/.
  • 43.http://www.cms.hhs.gov/ClinicalLabFeeSched/.
  • 44.https://www.noridianmedicare.com/p-medb/news/fees/2007/wa02/2007_wa02_index.html.
  • 45.Kulasingam SL, Kim JJ, Lawrence WF, Mandelblatt JS, Myers ER, Schiffman M, et al. Cost-effectiveness analysis based on the atypical squamous cells of undetermined significance/low-grade squamous intraepithelial lesion Triage Study (ALTS) J Natl Cancer Inst. 2006;98:92–100. doi: 10.1093/jnci/djj009. [DOI] [PubMed] [Google Scholar]
  • 46.Goldie SJ, Kim JJ, Wright TC. Cost-effectiveness of human papillomavirus DNA testing for cervical cancer screening in women aged 30 years or more. Obstet Gynecol. 2004;103:619–31. doi: 10.1097/01.AOG.0000120143.50098.c7. [DOI] [PubMed] [Google Scholar]
  • 47.Helms LJ, Melnikow J. Determining costs of health care services for cost-effectiveness analyses: the case of cervical cancer prevention and treatment. Med Care. 1999;37:652–61. doi: 10.1097/00005650-199907000-00005. [DOI] [PubMed] [Google Scholar]
  • 48.Shireman TI, Tsevat J, Goldie SJ. Time costs associated with cervical cancer screening. Int J Technol Assess Health Care. 2001;17:146–52. doi: 10.1017/s0266462301104137. [DOI] [PubMed] [Google Scholar]
  • 49.http://www.bls.gov/web/cpseed19.pdf.
  • 50.Insinga RP, Glass AG, Rush BB. Pap screening in a U.S. health plan. Cancer Epidemiol Biomarkers Prev. 2004;13:355–60. [PubMed] [Google Scholar]
  • 51.Mandelblatt JS, Lawrence WF, Womack SM, Jacobson D, Yi B, Hwang YT, et al. Benefits and costs of using HPV testing to screen for cervical cancer. J Natl Cancer Inst. 2002;287:2372–81. doi: 10.1001/jama.287.18.2372. [DOI] [PubMed] [Google Scholar]
  • 52.Harper DM, Raymond M, Noll WW, Belloni DR, Duncan LT, Cole BF. Tampon samplings with longer cervicovaginal cell exposures are equivalent to two consecutive swabs for the detection of high-risk human papillomavirus. Sex Transm Dis. 2002;29:628–36. doi: 10.1097/00007435-200211000-00003. [DOI] [PubMed] [Google Scholar]
  • 53.Brown DR, Shew ML, Qadadri B, Neptune N, Vargas M, Tu W, et al. A longitudinal study of genital human papillomavirus infection in a cohort of closely followed adolescent women. J Infect Dis. 2005;191:182–92. doi: 10.1086/426867. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Schiffman M. Integration of human papillomavirus vaccination, cytology, and human papillomavirus testing. Cancer. 2007;111:145–53. doi: 10.1002/cncr.22751. [DOI] [PubMed] [Google Scholar]
  • 55.Franco EL, Cuzick J, Hildesheim A, de Sanjose S. Chapter 20: Issues in planning cervical cancer screening in the era of HPV vaccination. Vaccine. 2006;24(Suppl 3):S171–7. doi: 10.1016/j.vaccine.2006.05.061. [DOI] [PubMed] [Google Scholar]
  • 56.Kulasingam SL, Myers ER. Potential health and economic impact of adding a human papillomavirus vaccine to screening programs. J Natl Cancer Inst. 2003;290:781–9. doi: 10.1001/jama.290.6.781. [DOI] [PubMed] [Google Scholar]
  • 57.Szarewski A, Cadman L, Mallett S, Austin J, Londesborough P, Waller J, et al. Human papillomavirus testing by self-sampling: assessment of accuracy in an unsupervised clinical setting. J Med Screen. 2007;14:34–42. doi: 10.1258/096914107780154486. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Waller J, McCaffery K, Forrest S, Szarewski A, Cadman L, Austin J, et al. Acceptability of unsupervised HPV self-sampling using written instructions. J Med Screen. 2006;13:208–13. doi: 10.1177/096914130601300409. [DOI] [PubMed] [Google Scholar]
  • 59.Hobbs MM, van der Pol B, Totten P, Gaydos CA, Wald A, Warren T, et al. From the NIH: Proceedings of a workshop on the importance of self-obtained vaginal specimens for detection of sexually transmitted infections. Sex Transm Dis. 2008;35:8–13. doi: 10.1097/OLQ.0b013e31815d968d. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES