Validation of self-reported history of colorectal cancer screening

Abstract

OBJECTIVE

This study aims to determine the validity of self-reported history of colorectal cancer (CRC) testing consisting of fecal occult blood tests (FOBTs)in the past 2 years or endoscopy (flexible sigmoidoscopy or colonoscopy) in the past 5 years by comparing it with reports provided by physicians.

DESIGN

A cross-sectional design was used for this study.

SETTING

Study participants were selected directly from the city’s population. Self-reported history of CRC testing was validated using records obtained from their physicians’ offices.

PARTICIPANTS

Participants were adults of 50 to 74 years, living within the boundaries of Calgary Health Region in Alberta.

INTERVENTIONS

Participants were recruited by a random-digit dial telephone survey of adults aged 50 to 74 years (n = 598). Following a phased process, a subset of these people (n = 200) agreed to provide names of their physicians to be contacted for their histories of CRC testing. Physicians’ reports were used to measure validity of self-reported history.

MAIN OUTCOME MEASURES

Agreement between self-reported history and physician’s records was measured using κ statistics and concordance. Validity of self-report was measured by calculating sensitivity, specificity, positive predictive values, and negative predictive values. Reasons for testing reported by the participants were compared with those reported by their physicians.

RESULTS

Complete information was received for 146 participants, revealing a 34.2% testing rate for CRC. Intermediate level of agreement for testing history (κ = 0.66 and concordance = 84.9%) was found between the 2 types of reporting for CRC testing. Self-reported history showed sensitivity of 76.0% (95% CI = 61.8%–86.9%) and specificity of 89.6% (95% CI = 81.7%–94.9%). High specificity was also observed for self-reporting of the individual tests, but low sensitivity was seen for the reporting of FOBT in the last 2 years. Most participants who correctly recalled the testing history also accurately identified the reason for testing (concordance = 80.0% for FOBT and 69.6% for endoscopy).

CONCLUSION

Self-reported history of CRC testing and physicians’ reports showed dependable agreement. Physicians need to probe their patients further for the history of FOBT. These results can be useful in clinical practice to determine the CRC screening status of the patients.

Self-report is the most common method of determining adherence to screening guidelines for various cancers. This method is widely used in clinical practice and research to determine whether patients have been screened for various cancers.^1,2 Validity of self-report varies with type, frequency, and duration of screening tests.^1,3 Such validity has been tested extensively for breast, cervical, and prostate cancers,^1,3–6 which showed high sensitivity and agreement. Few studies have tested the validity of self-report in assessing adherence to colorectal cancer (CRC) screening guidelines,^1,2,7 but all such studies have found patients to over-report the rates of screening tests. Only 1 study has reported on the validity of self-reported CRC screening in Canada’s publicly funded health care system, where the sample was drawn from the family members of CRC patients already enrolled with a provincial familial colon cancer registry.⁸ The current study assesses validity of self-reported screening and the reasons for testing in a general population sample.

Self-reported history provides useful information on compliance with screening guidelines and evaluating extent of screening programs.² Studies on the validity of self-reported history not only provide useful information to physicians and planners of health care in making appropriate decisions, but also change the way physicians communicate the need, process, and results of screening tests to their patients.¹

The Canadian Task Force on Preventive Health Care recommends annual or biennial screening with fecal occult blood tests (FOBTs) or flexible sigmoidoscopy for adults aged 50 and older.⁹ Other guidelines (such as those of the Canadian Association of Gastroenterology) recommend that all asymptomatic men and women over 50 years of age with no family history of CRC should undergo one of the following: FOBT every 2 years, flexible sigmoidoscopy alone or combined with FOBT every 5 years, double-contrast barium enema every 5 years or colonoscopy every 10 years.¹⁰

The objective of this study was to validate the self-reported screening history of CRC for adults aged 50 to 74 years, living in the Calgary Health Region in the province of Alberta.

METHODS

This validation study was embedded in a broader pilot test of a random-digit dial (RDD) telephone survey and an in-depth interview about CRC screening in Calgary Health Region. This health region has a total geographic area of 39 260 km² and a total population of 1 122 521 living predominantly in urban areas (approximately 84.5%).¹¹ A cross-sectional design was used to compare the self-reported CRC screening history with the records provided by the physicians. Data were collected in 3 steps. The first step consisted of RDD to recruit participants and obtain self-reports of CRC screening over the telephone. Participants were in the age group of 50 to 74 years, living within the boundaries of the Calgary Health Region. At the end of the RDD survey, participants were invited to take part in an in-depth interview conducted in person by trained interviewers. At the end of the in-depth interview, interviewers requested permission to contact participants’ physicians to obtain their histories of CRC testing. Consenting participants were asked to provide names and addresses of physicians they had visited within the past 6 years. Each identified physician was sent an explanatory letter, a copy of the signed consent form from the patient, and a form on which the physician could indicate the tests performed, reasons for testing, and the dates of the tests over the past 5 years. All the information was merged into a SAS¹² data set for data analysis.

Agreement between self-reported history and physician’s records was measured using κ statistics and by checking concordance between the 2 reports.^1,3 Validity of self-report was measured by comparing self-reported history with records provided by the physicians, and by calculating sensitivity, specificity, positive predictive values, and negative predictive values.^1,3 Finally, the reasons for testing reported by the participants were compared with those reported by their physicians.

The information provided by the participants during the RDD survey about any kind of testing (ie, FOBT in the last 2 years or endoscopy—sigmoidoscopy or colonoscopy—in the last 5 years) was compared with the information provided by the physicians for these tests in the same period. Comparisons were also made separately for FOBT within the last 2 years and endoscopy within the last 5 years. Prevalence rates of CRC testing for the entire population of 598 were recalculated based on the sensitivity and specificity results calculated in the validation study. This study was approved by the Conjoint Health Research Ethics Board at the University of Calgary.

RESULTS

The RDD telephone survey was conducted between June and August 2003, with 598 people aged between 50 and 74 years (response rate = 48.0%). Among the people surveyed, 52.7% (n = 315) agreed to receive information regarding the in-depth interview. A total of 221 in-person interviews were completed, giving a response rate of 70.2%. Among the people interviewed, 90.5% (n = 200) agreed to participate in the validation study. In total, 333 physicians were contacted (median = 1 physician per participant; range 1 to 6), of whom 246 returned completed forms (response rate = 73.9%). This resulted in 146 patient records with responses from all physicians identified by the participants available for the validation study. The final sample was demographically very similar to the original sample of 598 participants, but the people in the final sample were slightly more likely to have a usual source of care and to visit their physicians for regular physical examinations (Table 1).

Table 1.

Comparison of demographic characteristics between the overall and the final validation samples

SUBJECT CHARACTERISTICS	OVERALL SAMPLE (n = 598)		VALIDATION SAMPLE (n = 146)
	NUMBER	%	NUMBER	%
Age group
• 50–59	348	58.2	83	56.8
• 60–74	249	41.6	63	43.2
Sex
• Male	238	39.8	56	38.4
• Female	360	60.2	90	61.6
Residence
• In Calgary	475	79.4	115	78.8
• Outside Calgary	104	17.4	30	20.5
Education
• Less than high school	65	10.9	10	6.8
• High school diploma	106	17.7	20	13.7
• Trade certificate or diploma	213	35.6	51	34.9
• University	211	35.3	65	44.5
Self-rated health
• Excellent or very good	386	64.5	99	67.8
• Good	144	24.1	36	24.7
• Fair or poor	67	11.2	11	7.5
Have a usual source of care
• Yes	477	79.8	123	84.2
• No	121	20.2	23	15.8
Go for annual physical examination
• Yes	416	69.6	114	78.1
• No	182	30.4	32	21.9

Comparing self-reports with physicians’ records for any type of testing

According to physicians’ records, the overall prevalence of CRC testing (FOBT in the last 2 years or endoscopy in the last 5 years), was 34.2% (n = 50). Agreement on the reporting of CRC testing was found to be intermediate (κ = 0.66, 95% CI = 0.53–0.79, concordance = 84.9%). The overall sensitivity of participants’ responses, in comparison with physician records, was 76.0%, whereas the overall specificity was 89.6%. One reason for lower sensitivity could be “telescoping” (reporting a more recent date than the actual date for CRC screening tests). Telescoping was observed in 25% (3 in 12) of people who were classified as false positives for FOBT or endoscopy. Telescoping was also observed among 26% (10 in 38) of people who were classified as true positives because, despite reporting a more recent date for the test, both the self-reported date and the actual date fell within the set period of 2 years for FOBT and 5 years for endoscopy. Sensitivity and specificity for any CRC testing are shown in Table 2.

Table 2.

Accuracy of self-reported colorectal cancer screening history compared with medical records

SCREENING METHODS	NONRANDOM AGREEMENT		CONCORDANCE		SENSITIVITY		SPECIFICITY		POSITIVE PREDICTIVE VALUE		NEGATIVE PREDICTIVE VALUE
	κ SCORE	95% CI	%	95% CI	%	95% CI	%	95% CI	%	95% CI	%	95% CI
Any screening	0.66	0.53–0.79	84.9	78.1–90.3	76.0	61.8–86.9	89.6	81.7–94.9	79.2	65.0–89.5	87.8	0.79–0.93
Fecal occult blood test within past 2 years	0.47	0.28–0.65	84.2	77.3–89.7	55.6	35.3–74.5	90.8	84.1–95.3	57.7	36.9–76.6	90.0	83.2–94.7
Endoscopy within past 5 years	0.74	0.60–0.88	91.8	86.1–95.7	76.7	57.7–90.1	95.7	90.2–98.6	82.1	63.1–93.9	94.1	88.1–97.6

Comparing self-reports with physicians’ records for individual tests

According to physicians’ reports, the prevalence of FOBT in the past 2 years was 18.5% (n = 27), whereas the prevalence of endoscopy in the past 5 years was 20.6% (n = 30). Agreement between self-reports and physicians’ reports was tested for both FOBT in the last 2 years and endoscopy in the last 5 years (Table 2). Agreement on the reporting of FOBT was found to be low (κ = 0.47, 95% CI = 0.28–0.65, concordance = 84.2%). The reason for high concordance, despite low κ score, could be that most of the participants did not report any FOBT, which was confirmed by their physicians. The overall sensitivity of patients’ responses, in comparison with physician records, was 55.6%, whereas the overall specificity was 90.8%. Forty-four percent (12 in 27) of people reporting FOBTs were classified as false positives, resulting in low sensitivity for FOBT. False positives were more likely to be women (61.5% vs 28.6%, P = .08), and those with a high school education or less (53.8% vs 31.2%, P = .09). Reasons for the false positives were confusion with other CRC screening tests (6 in 12), unknown (5 in 12), or telescoping (1 in 12). Agreement on the reporting of endoscopy was found to be intermediate (κ = 0.74, 95% CI = 0.60–0.88, concordance=91.8%). The overall sensitivity of patients’ responses was 76.7%, whereas the overall specificity was 95.7%. Sensitivity and specificity for specific tests are shown in Table 2.

Measuring agreement on reasons for testing

Although limited by the low prevalence of CRC screening tests, we analyzed the agreement between reasons for testing reported by the participants and their physicians. These comparisons were made only for the participants who reported CRC screening that was validated by the physicians (true positives). Among the participants who correctly reported FOBT testing in the past 2 years, 80.0% (12 in 15) reported being tested for screening or because of positive family history. The self-reported reasons in this group also matched the reasons provided by the physicians 80.0% (12 in 15) of the time. Among those who correctly reported endoscopy testing in the past 5 years, 60.9% (14 in 23) of participants reported being tested for screening or because of positive family history. The self-reported reasons in this group matched the reasons provided by the physicians 73.9% (17 in 23) of the time. The reasons provided by the participants for any type of screening matched with the reasons provided by their physicians 76.3% (29 in 38) of the time.

DISCUSSION

Our comparison of patients’ self-reported history of CRC screening and physicians’ reports showed dependable agreement in the types of tests and concordance in the reasons for testing. However, assessment of validity showed low sensitivity and positive predictive values, which might be due to telescoping. This disparity can make it difficult for physicians and health care planners to decide future strategies at both the individual and the population levels. Our data could not find any specific demographic or other characteristics to distinguish people with high or low reliability for self-reporting of CRC testing, except that women and those with less education were more likely to provide false-positive information on having FOBT in the last 2 years. High concordance rates for the reason for conducting FOBT and colonoscopy also made self-reporting more reliable.

The major strength of this study is that the sample was drawn from the general population rather than from physicians’ offices or hospitals. A weakness of the study could be the low overall prevalence of CRC screening, which substantially reduced the number of people having particular tests and thus affected the power of the results. Another limitation could be the sex distribution of our sample, which differed from the population figures.¹¹

The effectiveness of physicians’ records as the standard for evaluating self-reported information in this study could also be questioned. Physician records have been used as standard for validating self-reported histories in several studies in Canada^13,14 and elsewhere.¹⁵ One study reported the accuracy of self-reported use of antidepressants among cancer patients by using physician records as standards.¹³ One more study from the Netherlands used physician records as a comparison to measure quality of cancer registry data.¹⁵ Another study, which compared patient and physician records for Papanicolaou smear histories, recommended that, if the physician response rate is high, physician records should be used as a standard to evaluate screening programs.¹⁴

These results strengthen our knowledge about the reliability of self-reporting of CRC screening. The results also indicate that the reliability of self-reporting for more invasive tests, such as sigmoidoscopy and colonoscopy, is higher than for FOBT, which is easier to confuse with other types of stool tests.³ The only study showing high sensitivity for FOBT (92.4%) was conducted by Mandelson, who actually did a 5-year follow-up study among women aged 50 to 79 years for FOBT only.⁴ This indicates that, by having a shorter follow-up time and focusing on more than 1 CRC screening test, patients might confuse both the timing and the types of tests.^1,3–6 Thus, it is important that physicians explain the test clearly while asking about the history of FOBT, and probe further about the exact timings of tests. Believing patient information about FOBT, without probing it further, might result in longer than recommended intervals for CRC screening.

The results also highlight the need to maintain up-to-date records of CRC screening, and to make that information accessible to other practitioners through a medium such as electronic health records. Availability of such information might help family physicians make appropriate decisions about educating patients and recommending CRC screening tests to their patients, in accordance with the available guidelines.

Conclusion

This study shows that self-reported history for CRC screening is reliable for testing status and reasons for testing. Physicians need to explain the test and to probe further for information about FOBT. Results of this study can help physicians improve their patients’ adherence to CRC screening guidelines.

EDITOR’S KEY POINTS

• Few studies have tested the validity of self-report in assessing adherence to screening guidelines for colorectal cancer (CRC), though all such studies have found that patients over-report the rates of screening tests.

• Results of the study indicate that the reliability of self-reporting for more invasive tests, such as sigmoidoscopy and colonoscopy, is higher than for fecal occult blood tests, which is easier to confuse with other types of stool tests. This highlights the need to maintain up-to-date records of CRC screening and to make that information accessible to other practitioners through a medium such as electronic health records.

POINTS DE REPÈRE DU RÉDACTEUR

• Peu d’études ont vérifié la validité de ce que les patients rapportent lorsqu’on évalue la fidélité aux directives pour le dépistage du cancer colo-rectal (CCR), et pourtant, toutes ces études trouvaient que les patients rapportent un taux excessif de tests de dépistage.

• Les résultats de cette étude indiquent que la fiabilité des rapports des patients pour les tests plus effractifs, tels la sigmoïdoscopie et la colonoscopie, est plus élevée que pour la recherche du sang occulte dans les selles, ce dernier test pouvant plus facilement être confondu avec d’autre types d’examens des selles. Cela souligne la nécessité de maintenir à jour les données de dépistage du CCR et de rendre cette information accessible aux autres intervenants via un médium tel que des dossiers de santé électroniques.