Inconsistencies Between Medical Records and Patient-Reported Recommendations for Follow-Up After Abnormal Pap Tests

Abstract

Purpose

Adherence with recommended follow-up after an abnormal Pap test is a critical step in the prevention of cervical cancer. Here, we focused on identifying inconsistencies between self-reported and health department record recommendations for follow-up.

Methods

Self-reported recommendations for follow-up were collected by questionnaire from 519 women with abnormal Pap tests in rural Appalachia as part of a trial of the efficacy of patient navigation. Health department medical records were reviewed to collect healthcare provider recommendations. Measures of inconsistency (discordance) were calculated for overall recommendations and each of three particular follow-up recommendations: repeat Pap test, referral for further tests, and other gynecologist referral.

Results

The inconsistencies between the recommendation from the health department records and self-reports ranged from 15.0% (repeat Pap test) to 35.3% (gynecologist referral). Inconsistencies were most common among women with a history of abnormal Pap tests and those with more severe initial results. Recommendations for repeat Pap tests were correctly reported most often when the women recalled receiving a letter stating the results. Of greatest concern were the inconsistencies regarding recommendations for referral to a gynecologist. The more severe the Pap test result, the greater the odds of inaccurate self-reports of receiving a referral to a gynecologist for follow-up, p<0.001.

Conclusions

Clinicians should be aware that patients with a history of abnormal results and severe Pap test abnormalities are at risk of misreporting recommendations for follow up.

Introduction

Results of approximately 6% of Pap tests performed each year in the United States are abnormal and require follow-up.¹ Adherence with recommended follow-up is a critical step in the prevention of cervical cancer. If follow-up care is obtained for abnormalities detected by Pap tests, nearly all cases of cervical cancer can be prevented or successfully treated. The role of healthcare providers includes informing women of abnormal Pap test results and communicating recommended follow-up. Understanding the type of follow-up that has been recommended is potentially one of the largest barriers to receiving appropriate care. A retrospective study of adherence to follow-up for abnormal Pap tests reported that misunderstanding the need for follow-up was one of the most commonly found barriers to the women receiving follow-up care.² If uncertainty about the need for follow-up is a barrier, confusion about the type of follow-up needed would also be a barrier to receiving the recommended follow-up care.

Relatively little research has been conducted on the inconsistencies between self-reported recommendations and health departments' medical records, and there is no known research focusing on rural health department patients. A number of studies have reported on the sources of error in self-report of cervical cancer screening,^3–6 but only one study⁷ was found that focused on the inconsistency of self-report of the recommended follow-up from an abnormal Pap test. Puleo et al.⁷ conducted medical chart audits and surveys with women who were members of four not-for-profit managed care plans. In this article, we report on the consistency of health department medical records and self-reported recommendations for follow-up after abnormal Pap tests among women receiving care in rural public health departments in Appalachian Kentucky.

Materials and Methods

We implemented a cluster randomized patient navigator intervention in 13 local health departments' cervical cancer screening programs in rural Kentucky. The project was approved by the IRBs of the University of Kentucky and the Kentucky Cabinet for Health and Family Services. The project was designed to test the effectiveness of patient navigation as a method to increase adherence with recommended follow-up for abnormal Pap tests. Women, 18 years and older, who had abnormal Pap test results were offered enrollment in the project. At enrollment, patient navigators recorded recommendations for follow-up noted by the health department in the woman's medical record. Recommendations included repeat Pap test, referral to a gynecologist, colposcopy, biopsy, other treatment, and other. Recommendations for follow-up were derived from the protocol of the Kentucky Women's Cancer Screening Program⁸ (KWSCP) a program of the National Breast and Cervical Cancer Early Detection Program (NBCCEDP).⁹ The terms used by the KWSCP to describe Pap test abnormalities include atypical squamous cells of undetermined significance (ASC-US), low-grade cervical squamous intraepithelial lesions (LSIL); atypical squamous cells, cannot rule out a high-grade lesion (ASC-H), and high-grade squamous intraepithelial lesion (HSIL).

Women also completed a written questionnaire that collected demographic characteristics and barriers to obtaining healthcare. To encourage and support women in obtaining recommended follow-up care, the patient navigators created an individualized navigation plan using information from the health department and the woman's questionnaire, focusing particularly on barriers to obtaining follow-up that were reported. As part of the questionnaire, each woman was asked what the health department had recommended she do for follow-up for the abnormal Pap test. The five possible responses were: Do nothing, just wait; Repeat the test; Schedule other follow-up tests; They referred me to another doctor; and Other. In both the health department record and the participant questionnaire, more than one answer could be selected if multiple recommendations were given.

Measures of agreement and analysis

To facilitate analysis, health department recommendations and self-reported recommendations were reclassified. For health department recommendations, colposcopy, biopsy, and other treatment were reclassified to: Further tests. Responses to the Other category were reviewed individually and reclassified into one of the three categories as appropriate, with the exception of two responses that we were not able to reclassify. Self-reports of recommendations were reclassified in a similar manner. As with the health department recommendations, all Other responses were reviewed and reclassified as appropriate. Only five (1%) responses were unable to be reclassified.

Binary outcome measures of discordance (inconsistency) between the health department records and self-reports were calculated for each of three recommendation categories: repeat the Pap test, further tests, and referral to gynecologist. For each of these recommendations, the responses from the participant questionnaire and health department record were dichotomized into whether or not they concurred. If responses disagreed on the particular recommendation, they were considered discordant for that recommendation. As <3% (n=14) of the subjects answered Do nothing, 1% (n=5) answered Other, and only 0.5% (n=2) of health department records did not list any recommendations beyond Other, discordance was not calculated independently as an outcome for Do nothing or Other. However, these participants were not excluded from the analyses.

Overall concordance was defined as agreement between the health department record and self-report. When more than one recommendation was present in the health department record, overall concordance required agreement for all recommendations. Kappa agreement statistics¹⁰ were calculated for each of the recommendations to assess the concurrence of the patient navigator and the participant self-reports. Because the study design was a cluster randomization scheme, the association of the binary measures of inconsistency with demographic covariates (including potential barriers to follow-up) was assessed using multivariable generalized estimating equation (GEE) models that adjust for the clustering of women within the local health departments. Estimated intraclass correlations for each model along with the odds ratios (OR) and corresponding p values are presented.

Results

A total of 519 women were enrolled in the project across 13 health departments. Table 1 summarizes the demographic characteristics of the study population. The majority were Caucasian between the ages of 18 and 44, over two thirds had completed high school, household income was quite low, and >50% of participants did not have health insurance. However, the study population differed from the general age-specific population in terms of marital status, as a majority of the enrolled women were unmarried. Distribution of the type of Pap test abnormalities reported was as follows: ASC-US, 217 (41.8%); LSIL, 248 (47.8%); and ASC-H or HSIL, 54 (10.4%). Of the 519 participants, 14 (2.7%) reported they were told they needed no follow-up (Do nothing), 175 (33.7%) reported they needed a repeat Pap test, 215 (41.4%) reported they needed to schedule further tests or treatment, 127 (24.5%) reported they were referred to a gynecologist for follow-up, and 5 (1.0%) reported a recommendation not listed (Other). As noted previously, subjects could select multiple responses as well as the patient navigators if the health department records documented multiple recommendations. Over 40% of participants had a history of previous abnormal Pap test results.

Table 1.

Participant Characteristics

Characteristic	n	%
Age
18–24	251	48.4
25–44	206	39.7
45+	62	12.0
Education
Less than high school	115	22.2
High school graduate	249	48.0
Post-high school	155	29.9
Income
<$10,000	213	41.0
$10,000–$50,000	241	46.4
$50,000+	18	3.5
Unknown/not reported	47	9.1
Married
Yes	173	33.3
No	343	66.1
Unknown/not reported	3	0.6
Race
White	499	96.2
Othera	20	3.8
Employed
Yes	240	46.2
No	277	53.4
Unknown/not reported	2	0.4
Insurance
Private	79	15.2
Medicare/Medicaid	140	27.0
None	296	57.0
Unknown/not reported	4	0.8
How told
Call	218	42.0
Letter	184	35.5
Call and letter	85	16.4
Otherb	29	5.6
Unknown/not reported	3	0.6
Pap results
ASC-US	217	41.8
LSIL	248	47.8
ASC-H/HSIL	54	10.4
History of abnormal Pap test
Yes	208	40.1
No	306	59.0
Unknown/not reported	5	1.0
Transportation Issues
Yes	79	15.2
No	440	84.8

^aOther race responses: 5 black/African American, 2 Hispanic, 1 Native Hawaiian, 1 biracial, and 11 unspecified.

^bOther how told responses: 18 visits to health department (HD), 4 participants called HD, 5 call from other than HD, 4 unspecified.

ASC-H, atypical squamous cells, cannot rule out a high-grade lesion; ASC-US, atypical squamous cells of undetermined significance; HSIL, high-grade squamous epithelial lesion; LSIL, low-grade epithelial lesion.

Table 2 summarizes the extent to which the self-reported recommendation for follow-up from participants agreed with recommendations noted by the health departments. As Table 2 shows, the kappa statistics for agreement vary across the possible recommendations. Self-reports agreed moderately with the health department recommendations for needing a repeat Pap test (κ=0.69) and scheduling further tests (κ=0.60). However, agreement was low with respect to referring participants to another gynecologist (κ=0.28).

Table 2.

Participants' Knowledge

	Patient reported repeat Pap needed
	No	Yes	Kappa	Sensitivity	Specificity	PPV	NPV
HD reported repeat Pap needed
No	276 (53.2%)	10 (1.9%)	0.69	0.71 (0.67–0.75)	0.97 (0.95–0.98)	0.80 (0.77–0.84)	0.94 (0.92–0.96)
Yes	68 (13.1%)	165 (31.8%)

	Patient Reported further tests needed
	No	Yes
HD reported colposcopy, biopsy, or treatment needed
No	224 (43.2%)	24 (4.6%)	0.60	0.70 (0.67–0.74)	0.90 (0.88–0.93)	0.74 (0.70–0.77)	0.89 (0.86–0.92)
Yes	80 (15.4%)	191 (36.8%)

	Patient reported referral to gynecologist needed
	No	Yes
HD reported referral to gynecologist needed
No	241 (46.4%)	32 (6.2%)	0.28	0.39 (0.34–0.43)	0.88 (0.86–0.91)	0.61 (0.57–0.66)	0.75 (0.71–0.79)
Yes	151 (29.1%)	95 (18.3%)

Do nothing and Other are not represented as separate outcomes; however, those responses were not dropped from analysis. (n=519 for each subsection of table.)

NPV, negative predictive value; PPV, positive predictive value.

The inconsistencies between the recommendation from the health department records and self-reports was 15.0% for repeat Pap test, 20.0% for further tests, and 35.3% for gynecologist referral (Table 2). The overall disagreement for follow-up recommendations was 53.8%. The percent of women incorrectly reporting no need for a gynecologic referral was twice as high as those incorrectly reporting no need for a repeat Pap test, which is supported by the kappa statistics (0.69 vs. 0.28). For all recommendations, most inconsistencies (77%–87%) were false negatives, reports of not needing follow-up. These instances of false negative reports were recorded in 13.1% of participants for repeat Pap test, 15.4% for further tests, and 29.1% for gynecologist referral. False positives, that is, reporting needing follow-up when it was not noted in the health department medical record, accounted for only 1.9%–6.2% of inconsistencies, depending on the type of follow-up.

Measures of sensitivity and specificity of the self-report of recommended follow-up actions are also shown in Table 2. Self-reports of all three types of recommendations were more specific than sensitive. The self-report of a repeat Pap test recommendation was highly specific (0.97), with a sensitivity of 0.71. Similarly, self-reports of further tests needed were slightly lower (sensitivity=0.70, specificity=0.90). The specificity and sensitivity of self-reports of a gynecologist referral were the lowest of the three types of recommendations (specificity=0.80, sensitivity=0.39).

Table 3 summarizes the relationship between selected demographic variables and the percent of inconsistent reports by type of follow-up recommendations from the multivariable models.

Table 3.

Selected Demographics and Discordance

	Overall discordance			Repeat Pap test discordance			Further tests/treatment discordance			Referral to gynecologist discordance
	ICC=0.0708			ICC=0.0544			ICC=0.0444			ICC=0.1972
Variable	%	OR	p value	%	OR	p value	%	OR	p value	%	OR	p value
Age
18–24	47.4	0.54	0.19	12.8	0.36	0.007	17.9	0.76	0.56	31.1	0.99	0.97
25–44	59.2	0.87	0.70	14.1	0.39	0.04	21.4	0.74	0.34	42.2	1.60	0.06
45+	61.3	1.00		27.4	1.00		24.2	1.00		29.0	1.00
Education
Less than high school	55.7	1.86	0.03	14.8	1.40	0.34	20.9	1.33	0.29	33.9	1.27	0.40
High school graduate	47.4	1.00		13.3	1.00		16.5	1.00		32.9	1.00
Post-high school	62.6	1.90	0.009	18.1	1.18	0.21	25.2	1.41	0.14	40.0	1.61	0.14
Income
<$10,000	57.3	1.00		15.0	1.00		24.9	1.00		36.2	1.00
$10,000–$50,000	53.1	0.92	0.68	15.4	1.14	0.62	18.7	0.75	0.28	35.3	0.98	0.93
$50,000+	55.6	0.95	0.92	27.8	3.24	0.06	22.2	0.80	0.71	33.3	0.69	0.47
Married
Yes	48.0	0.74	0.03	14.5	0.70	0.08	13.9	0.62	0.03	33.5	0.93	0.70
No	56.9	1.00		15.5	1.00		23.3	1.00		36.2	1.00
Employed
Yes	52.9	0.74	0.29	14.6	0.63	0.24	17.5	0.55	0.04	35.4	1.02	0.95
No	54.2	1.00		15.2	1.00		22.0	1.00		35.0	1.00
Insurance
Private	62.0	1.00		24.1	1.00		25.3	1.00		38.0	1.00
Medicare/Medicaid	47.9	0.53	0.06	9.3	0.31	0.09	19.3	0.46	0.003	32.1	0.85	0.59
None	54.4	0.61	0.20	15.5	0.79	0.65	19.3	0.51	0.07	35.8	0.70	0.17
How told
Call and letter	65.9	1.35	0.45	16.5	0.93	0.85	22.4	0.82	0.55	48.2	1.43	0.05
Call	58.3	1.00		16.5	1.00		27.1	1.00		35.3	1.00
Letter	42.9	0.51	0.04	11.4	0.60	0.06	11.4	0.46	0.02	29.4	0.61	0.11
Other	55.2	1.11	0.85	20.7	2.17	0.07	17.2	0.86	0.76	37.9	0.86	0.83
Pap results
ASC-US	42.9	1.00		17.1	1.00		15.7	1.00		24.9	1.00
LSIL	59.3	2.31	<0.0001	13.7	0.96	0.89	23.8	1.57	0.04	39.9	2.05	<0.0001
ASC-H/HSIL	72.2	2.50	0.01	13.0	0.56	0.22	20.4	1.12	0.80	55.6	3.20	0.0004
History of abnormal Pap
Yes	59.6	1.57	0.03	14.4	0.83	0.30	23.1	1.19	0.54	40.4	1.60	0.02
No	49.4	1.00		15.0	1.00		18.0	1.00		31.7	1.00
Transportation issues
Yes	51.9	0.85	0.56	12.7	0.68	0.30	15.2	0.54	0.10	36.7	1.35	0.25
No	54.1	1.00		15.5	1.00		20.9	1.00		35.0	1.00

ICC, intraclass correlation; OR, odds ratio.

Inconsistencies, overall

Self-reports from participants with a high school diploma or GED were more likely to be consistent with health department records than were those of other participants (OR 1.86 and 1.90, p=0.03 and 0.009). Self-reports from women with a history of abnormal Pap tests were more likely to disagree with the health department records than were those of women with their first abnormal Pap test result (OR 1.57, p=0.03). Inconsistencies were also found to be related to the type of Pap test result. Self-reports and health department records among those with LSIL were more than twice as likely to be inconsistent as were those with ASC-US (OR 2.31, p<0.0001). Similarly, those with even more severe results of ASC-H or HSIL were 2.5 times as likely to be inconsistent (p=0.01). In general, the more severe the Pap test result, the more likely self-report and health department records were to disagree.

Inconsistencies, repeat Pap test

Reports from women > age 44 were more likely to be inconsistent with health department recommendations than were reports from younger women about the need for a repeat Pap test (OR 0.36 and 0.39, p=0.007 and 0.04). Only 11% of women who self-reported recommendations for a repeat Pap test and were informed of their abnormal result in a letter disagreed with the health department record, which is significantly less than among those who were told by means other than a call or letter (OR 0.60 vs. 2.17, p=0.005).

Inconsistencies, further testing

Self-reports of recommendations for further testing were almost twice as likely to be inconsistent with health department recommendations for unmarried women as for married or partnered women (OR 0.62, p=0.03). Likewise, women who were not employed were also almost twice as likely as employed women to be inconsistent with health department records in reporting recommendations for further testing (OR 0.55, p=0.04). Women with Medicare/Medicaid or no insurance were more likely to be consistent with health department recommendations than were women with private insurance (OR 0.46 and 0.51, p=0.003 and 0.07). Women who were informed by a letter of their abnormal Pap test results were over twice as likely to report consistent recommendations as were those who were informed by a phone call (OR 0.46, p=0.02). Almost 24% of participants with LSIL incorrectly reported their need for further tests, which is a significantly greater percentage than that among participants with ASC-US (OR 1.57, p=0.04).

Inconsistencies, referral to a gynecologist

Only 60% of women with an initial result of LSIL correctly reported a referral to the gynecologist. Of those with ASC-H or HSIL, less than half correctly reported having been referred to a gynecologist. Hence, patients with more severe Pap test results are more likely to inaccurately report a referral to a gynecologist for follow-up (OR 2.05 and 3.20, p<0.0001 and 0.0004). Participants with a history of abnormal Pap tests were over 1.5 times as likely as those without such a history to misreport a recommendation to obtain follow-up from a gynecologist (p=0.02).

Discussion

Overall, <50% of women in our study correctly reported follow-up recommendations that agreed with the health department records. The most commonly misreported recommendation was a referral to a gynecologist, and only 15%–25% of the inconsistent reports were false positives, that is, a participant falsely reporting needing a particular follow-up when the health department record did not. Puleo et al.⁷ surveyed women on agreement of self-reported recommendations from cancer screening and medical records. They found differing false positive results for cervical cancer screening; with respect to further testing, 3%–18% of the responses were false positives. However, for both repeat Pap test in <6 months and gynecologist referral, 60%–66% of the inconsistencies reported by the participants were false positives. In our study, almost all the inconsistencies were false negatives; women reported they did not need a particular type of medical follow-up when according to the health department records, they did. The false negatives are more worrisome, especially in women with the most severe results from the original abnormal Pap test. Puleo et al.⁷ also found a negative association between the abnormal Pap result and the agreement between health department records and self-report.

In general, the more time that elapses between an abnormal Pap test and subsequent follow-up, the greater risk of needing more invasive treatment and a worse prognosis.^11–13 If a woman does not understand or recall recommendations, she may delay taking action to obtain follow-up or even ignore the recommendation completely,² which in turn can increase her risk of developing cervical cancer.

Leyden et al.¹² studied women diagnosed with invasive cervical cancer who had a failure in follow-up. They found these women were almost twice as likely to be residents of low-education areas (<25% high school educated) and of high poverty areas (>20% below the federal poverty level). The educational level of our study population was higher than the levels reported in the Leyden et al. study, and we found that self-reports from women with a high school diploma or GED were more likely to be consistent with the health department records than were self-reports of women with either less than a high school diploma or GED or post-high school education. The pattern of adherence by educational level that we found is similar to findings reported by other investigators.^3,7,11 We expected education to correlate more directly with consistency, and our findings led us to conclude that additional research is needed on the role of educational attainment in Appalachian populations in explaining adherence with recommendations for follow-up. We did not find any differences in agreement by income even though the study population is located in a high-poverty area.

Of greatest concern is the relationship between the severity of the Pap test results and the reported awareness of the recommendation given. A previous study by Eggleston et al.¹¹ found that white women with more severe Pap test results were least likely to be adherent to the recommended follow-up. In our study, women with ASC-US were significantly more likely to correctly report needing follow-up than were women with more severe diagnoses for both the overall recommendation and the referrals to a gynecologist. Our findings suggest that adherence may be lower as a result of misunderstandings about the particular follow-up being recommended.

Puleo et al.⁷ reported sensitivity measures of 0.86 of the patient-reported follow-up vs. the medical record for both recommendations for a repeat Pap test within 6 months and a referral to see another doctor. They also report sensitivity and specificity for recommendations for specific follow-up testing. For colposcopy, they report 0.92 and 0.95 (sensitivity and specificity, respectively), and for more invasive procedures, such as biopsy and loop electrosurgical excision procedure (LEEP), they report 0.78 and 0.99. Our results indicated lower sensitivity but greater specificity, including 0.97 for a repeat Pap test with no associated time frame and 0.88 for a gynecologist referral. We found measures of 0.70 and 0.90 (sensitivity and specificity) for recommendations for further testing.

The women in the Puleo et al. study⁷ were all members of a managed healthcare organization, whereas over half of the women enrolled in our study were uninsured and received care through the Kentucky Women's Cancer Screening Program.⁸ For many of the women in our study, a referral to a gynecologist was difficult to accomplish because of the additional cost burden placed on the participant or the lower accessibility to healthcare providers in rural Appalachia. As part of the baseline participant questionnaire in our study, women were asked about barriers to receiving follow-up care for an abnormal Pap test. Over 45% of the women chose as a barrier to obtaining follow-up care: It costs too much. This finding is consistent with findings in other studies that have found lack of insurance and money to be barriers to receiving recommended follow-up care.² Given that almost half of the women in our study indicated that cost was important to them, financial considerations are likely to be key components in making decisions about obtaining follow-up care for rural, low-income populations. Interestingly, we found that self-reports from participants with private insurance were consistently less likely to agree with health department records. A possible explanation for this finding is that those with private insurance are more likely to have out-of-pocket expenses for copays and deductibles than are those whose care is paid for with public funds.

Participants who received their Pap test results in a written form instead of by phone call were more likely to agree with the health department recommendations. This suggests that recommendations may be more likely to be understood when provided in written form. Limited health literacy notwithstanding, written recommendations may be less subject to distortion than telephone calls or other less formal methods of communication. McKee et al.¹³ reported that women who were notified by letter of the results of their abnormal Pap tests were significantly more likely to have been adherent and to have received the recommended colposcopy. If a participant receives a letter describing the abnormal Pap test results, the letter can be used for reference when contacting the health department about the abnormal result and the recommended follow-up. Our study adds to the current body of literature on adherence with follow-up for abnormal Pap tests in reporting on Appalachian populations; there are few reports of adherence with recommended follow-up in this rural, low-income population.^4,11 Most studies focusing on adherence among lower socioeconomic status populations have reported on urban areas.^3,5,13

The results from this study should be interpreted with caution, as there were limitations to the research methods and data collection. The recording of recommendations for follow-up care in the patient's record differed among health departments. Accordingly, recoding was necessary to develop analysis files, and some loss of data may have occurred.

Although the overall concordance between the recommendations noted by the patient navigator from the health department record and the recommendations reported by the participant were high, results in some subgroups were inconsistent. In particular, providers must be aware of the barriers women have in understanding recommendations. Clinicians should be aware that older women, women with a history of abnormal Pap test results, and women with ASC-H/HSIL Pap test results may be more likely to underreport/underestimate the severity of their condition and downplay the need for further follow-up. Consequently, clinicians should focus on recognizing the potential for misunderstanding of recommendations for follow-up for this subset of patients and should strive to provide education and recommendations with written correspondence as backup.

Acknowledgments

This project was supported by a grant from the National Cancer Institute (R01 CA 120606). Support for data management was provided by Katherine L. Jones.

Disclosure Statement

No competing financial interests exist.