Correlates of self-reported colorectal cancer screening accuracy in a multi-specialty medical group practice

Abstract

Purpose

We assessed whether accuracy of self-reported screening for colorectal cancer (CRC) varied by respondent characteristics or healthcare utilization.

Methods

From 2005 to 2007, 857 respondents aged 51 – 74 were recruited from a multi-specialty medical group practice to answer a questionnaire about their CRC screening (CRCS) behaviors. Self-reports were compared with administrative and medical records to assess concordance, sensitivity, specificity, and report-to-records ratios for overall CRCS (fecal occult blood test, sigmoidoscopy, and/or colonoscopy).

Results

Concordance was good (≥0.8 to <0.9) or fair (≥0.7 to <0.8) for most subgroups; respondents with >5 visits outside the clinic had poor (<0.7) concordance. Sensitivity estimates were mostly excellent (≥0.9) or good but poor for respondents whose healthcare provider did not advise a specific CRCS test. Specificity was poor for the following respondents: 65+ years, males, college graduates, family history of CRC, >5 visits outside of the clinic, or whose healthcare provider advised a specific CRCS test. Respondents 65+ years and with >5 outside visits over-reported CRCS.

Conclusions

With few exceptions, self-reports of CRCS in an insured population is reasonably accurate across subgroups. More work is needed to replicate these findings in diverse settings and populations to better understand subgroup differences and improve measures of CRCS.

1. INTRODUCTION

Although colorectal cancer screening (CRCS) rates are increasing [1], there is still room for improvement if we are to achieve Healthy People 2010 goals [2] and eliminate disparities. Monitoring adherence to guidelines enables us to assess progress towards meeting these goals and to identify screening disparities for population subgroups [2]. Adherence to CRCS guidelines is often assessed using self-reported data, in part, because of the time, cost, and limited access to medical records [3]. Increased reliance on self-reports underscores the need for accurate measures of adherence [3,4].

Although a number of studies have assessed agreement between self-reported CRCS and administrative data or medical records [5–19], fewer have examined sub group differences in accuracy [5,6,8–12,15,16,19], and- most were limited to socio-demographic characteristics.

No studies have examined whether healthcare utilization factors, such as the number of visits to a healthcare provider, is associated with accuracy of self-reported CRCS. Identifying subgroup differences in the accuracy of self-reported CRCS may assist in the interpretation of prevalence estimates from survey data and the results of behavioral interventions. Understanding differences also may be useful in guiding clinical decision-making and improving patient-physician communication about CRCS.

We used data from a randomized controlled trial designed to evaluate the reliability and validity of a standardized self-report questionnaire of CRCS behaviors to examine whether the accuracy of self-report measures of CRCS behavior varied by respondent characteristics and healthcare utilization.

2. MATERIALS AND METHODS

2.1. Study Setting and Sample Selection

A questionnaire, developed by one of the authors (SWV) in collaboration with scientists at the National Cancer Institute (NCI) and the Centers for Disease Control and Prevention (CDC) [4,12], was evaluated for reliability and validity using three modes of survey administration mail, telephone, and face-to-face [13]. Study participants were men and women, 51 to 74 years old, who were primary care patients for at least 5 years at a large multispecialty medical group practice in Houston, Texas [13]. Patients with a prior history of CRC were excluded. From 2005 to 2007, 1004 patients were randomized to mail, telephone, or face- to-face mode of survey administration. Of these, 857 completed a baseline survey and were included in this analysis. Additional details about recruitment, eligibility, study design and study procedures are described elsewhere [13]. The study protocol was approved by the University of Texas Health Science Center at Houston Committee for Protection of Human Subjects.

2.2. Measures

Adherence to CRCS guidelines was defined as: a fecal occult blood test (FOBT) within the past year, flexible sigmoidoscopy within the past 5 years, or colonoscopy within the past 5 years. For FOBT and sigmoidoscopy, these recommendations are the same as the American Cancer Society (ACS) guidelines in effect at the time of the study [20]. Instead of colonoscopy within the past 10 years per ACS guidelines, we restricted the measure to within the past 5 years to match our eligibility criteria. This ensured a sufficient number of patients and reduced the likelihood of receiving CRCS from an outside provider.

Self-reports were compared to a combined database of administrative and medical records (referred to as the combined medical record). Type of tests and dates of each test were abstracted from the combined medical record. Inter-rater agreement was assessed for three pairs of raters for 81 patients. For the most recent test within guidelines, agreement was 98% (kappa = 0.96). For all tests within the past 5 years, agreement was 91% (kappa = 0.89). If a patient reported a test from an outside provider that was not recorded in the combined medical record, we contacted the provider to confirm the report [13]. Of the 30 providers contacted, 23 provided the requested information, and this information was added to the database.

We assessed the following characteristics through the survey: age (categorized as 51 – 64, 65+); gender (male, female); race/ethnicity (non-Hispanic white, African American, other); marital status (not married, married/living with partner); education (<high school diploma/General Equivalency Diploma (GED), some college, college+); family history of cancer (yes, no); number of physician visits in the past 5 years at the clinic (0 – 5, >5); number of physician visits in the past 5 years outside of the clinic (0 – 5, >5); and whether their healthcare provider advised a specific CRC test (FOBT, sigmoidoscopy, colonoscopy). We included respondent characteristics, CRC history and healthcare provider advised a specific CRC test to compare our results with previous studies.

2.3. Data Analysis

We combined data from the three survey modes because there were no differences in validity estimates by mode [13]. We computed concordance (i.e., percentage agreement), sensitivity, specificity, and report-to-records ratio for any CRCS test (FOBT, sigmoidoscopy and/or colonoscopy) by subgroup. Subgroups with less than 5 in a cell for their respective 2 × 2 tables were collapsed into an adjacent category if possible; those subgroups that could not be collapsed were excluded. Two-sided 95% confidence intervals were calculated for all measures. Analyses by each CRC test were attempted but yielded unstable estimates for multiple subgroups because cells (<5) could not be meaningfully collapsed.

Tisnado et al.’s [21] criteria for evaluating sensitivity and specificity of ambulatory care services were used to assess concordance, sensitivity, and specificity: >0.9 denotes excellent agreement, >0.8 to <0.9 denotes good agreement, >0.7 to <0.8 denotes fair agreement and <0.7 denotes poor agreement. The report-to-records ratio is a measure of net bias in test reporting [22]. Values >1.0 indicate over-reporting, and values <1.0 indicate under-reporting; 95% confidence intervals were used to determine the precision of over- or under-reporting. All data were analyzed using Microsoft Excel 2007 and Inter-cooled Stata version 9.0 (Stata Corporation College Station, TX).

3. RESULTS

Most respondents were 51 – 64 years old (81.2%), female (65.8%), married or living with a partner (74.2%), and college educated (54.6%); 59.4% were non-Hispanic white and 26.4% were African American. Approximately 11.3% had a family history of CRC. Most (93.2%) had a clinic visit within the last year, and 86.8% had more clinic visits within the past 5 years. Only 14.5% had more than 5 visits outside the clinic within the past 5 years. Most reported having had FOBT, sigmoidoscopy and/or colonoscopy in the past (61.5%).

Concordance was mostly good, except for the following subgroups: respondents aged 65 years and older, those with a family history of CRC, those with 5 or less visits to the clinic, and those with more than 5 visits outside the clinic (Table 1). For sensitivity, estimates were excellent or good except for respondents whose healthcare provider did not advise a specific CRCS test. In contrast, specificity estimates were mostly fair but were poor for several subgroups, including those aged 65 years and older, male, college graduates, positive family history of CRC, more than 5 visits to an outside provider in the past 5 years, and whose provider advised a specific CRCS test. Report-to-records ratios indicated over-reporting for respondents aged 65 years and older and those with more than 5 visits to an outside provider within the past 5 years.

Table 1.

Concordance, sensitivity, specificity, and report-to-records ratio comparing self-report of colorectal cancer screening within guidelines^* to combined medical record (“gold standard”) by respondent characteristics.

		Estimate (95% confidence interval)
	n	Concordance	Sensitivity	Specificity	Report-to-record ratio
Age
51 – 64	696	0.84 (0.81 – 0.87)	0.92 (0.88 – 0.95)	0.74 (0.69 – 0.80)	1.11 (0.80 – 1.42)
65+	161	0.76 (0.68 – 0.83)	0.87 (0.79 – 0.95)	0.69 (0.58 – 0.80)	1.37 (1.06 – 1.68)
Gender
Male	293	0.80 (0.75 – 0.85)	0.92 (0.87 – 0.97)	0.69 (0.60 – 0.78)	1.25 (0.94 – 1.56)
Female	564	0.84 (0.81 – 0.87)	0.91 (0.86 – 0.95)	0.76 (0.69 – 0.82)	1.09 (0.79 – 1.40)
Race/ethnicity
Non-Hispanic white	509	0.82 (0.79 – 0.86)	0.93 (0.89 – 0.97)	0.72 (0.65 – 0.78)	1.21 (0.90 – 1.52)
African American	226	0.83 (0.78 – 0.89)	0.91 (0.84 – 0.97)	0.73 (0.63 – 0.83)	1.11 (0.80 – 1.42)
Other#	122	0.84 (0.76 – 0.91)	0.88 (0.77 – 0.99)	0.77 (0.63 – 0.90)	1.03 (0.72 – 1.33)
Marital status
Not married	215	0.85 (0.79 – 0.90)	0.93 (0.88 – 0.98)	0.77 (0.68 – 0.86)	1.19 (0.88 – 1.50)
Married/living with partner	636	0.82 (0.79 – 0.85)	0.90 (0.86 – 0.95)	0.71 (0.65 – 0.78)	1.13 (0.82 – 1.44)
Education
High school/GED or less	115	0.83 (0.76 – 0.91)	0.89 (0.80 – 0.98)	0.78 (0.66 – 0.90)	1.13 (0.82 – 1.43)
Some college	267	0.83 (0.78 – 0.88)	0.90 (0.85 – 0.96)	0.76 (0.68 – 0.84)	1.14 (0.83 – 1.45)
College graduate	468	0.82 (0.78 – 0.86)	0.92 (0.87 – 0.96)	0.69 (0.61 – 0.77)	1.15 (0.84 – 1.46)
Family history of colorectal cancer
No	736	0.83 (0.80 – 0.86)	0.92 (0.88 – 0.95)	0.74 (0.68 – 0.79)	1.14 (0.83 – 1.45)
Yes	97	0.78 (0.69 – 0.88)	0.88 (0.75 – 1.00)	0.66 (0.48 – 0.84)	1.13 (0.82 – 1.43)
No. visits to KSC in past 5 years
0 – 5	112	0.79 (0.71 – 0.88)	0.87 (0.77 – 0.97)	0.73 (0.60 – 0.86)	1.17 (0.86 – 1.48)
>5	736	0.83 (0.80 – 0.86)	0.92 (0.88 – 0.95)	0.73 (0.67 – 0.78)	1.14 (0.83 – 1.45)
No. visits outside KSC in past 5 years
0 – 5	725	0.85 (0.82 – 0.88)	0.91 (0.88 – 0.95)	0.77 (0.71 – 0.82)	1.10 (0.79 – 1.41)
>5	123	0.69 (0.59 – 0.79)	0.88 (0.78 – 0.98)	0.56 (0.40 – 0.71)	1.51 (1.20 – 1.82)
Healthcare provider advised a specific CRCS test
No	113	0.88 (0.82 – 0.95)	0.56 (0.45 – 0.66)	0.95 (0.90 – 0.99)	0.83 (0.53 – 1.14)
Yes	744	0.82 (0.79 – 0.85)	0.93 (0.89 – 0.96)	0.66 (0.60 – 0.73)	1.16 (0.85 – 1.46)

^*Adherence to guidelines is defined as any CRCS test within guidelines (annual fecal occult blood test, sigmoidoscopy within past 5 years, and/or colonoscopy within past 5 years). A concordance, sensitivity, or specificity of >0.9 denotes excellent agreement, >0.8 to <0.9 denotes good agreement, >0.7 to <0.8 denotes fair agreement and <0.7 denotes poor agreement (21). Missing values were excluded so totals may not add up to 857. GED = General Equivalency Diploma; KSC = Kelsey Seybold Clinic; CRCS = colorectal cancer screening;

^#Includes Hispanic, Asian/Pacific Islander, Native American and multi-racial groups.

4. DISCUSSION

Our finding that older patients over-reported any CRCS test within guidelines was similar to Partin et al.’s [12], suggesting that researchers and providers may need to rely on other sources of information to ascertain screening history for this subgroup. Other published studies on correlates of accuracy of self-reported CRCS tests are limited primarily to socio-demographics and show little consistency in the validity measures reported [5,6,8–12, 19]. Six studies used one or more of the four validity measures as our study-concordance [8–12,19], sensitivity [8–10,12,19], specificity [8–10,12,19] and/or report-to- records ratio [12] to determine whether age [10–12], race/ethnicity [8,9,11,12,19], sex [9,12], education [11,12], marital status [12] and family history of CRC [11,12] were associated with accuracy of self-reported FOBT [8–11,19], sigmoidoscopy [8,9,19], colonoscopy [8,19] and/or any CRCS [12]. Inconsistencies in findings among these studies [8–10,12,19] are likely due to variation in the populations studied, the time interval used to assess recall, and the CRCS guidelines used to measure validity which limited comparability of our findings to only one study [12]. Study samples included carpenters [10], patients from health maintenance organizations [8,9], primary care clinics [19] and users of the Veteran’s Administration healthcare facilities [12]. The time interval used to compare self-reported CRCS behaviors with medical records also differed across studies, particularly for FOBT where the time intervals included one [10,12,19], two [9], and five years [8]. Only two studies [12,19] used evidence-based guidelines to assess validity of self-reported CRCS.

To our knowledge, this is the first study to examine whether accuracy of self-reported CRCS varied by healthcare utilization. Patients who had more than 5 visits to an outside provider in the past 5 years may over-report CRCS. For patients who reported seeing outside providers, we were able in most instances to contact those providers and verify self-reports; however, patients may not have provided complete information about the medical care they received elsewhere which might explain why specificity and concordance were poor. Nevertheless, providers need to be diligent in taking medical histories among patients using multiple sources of medical care. Further, there is a need for better patient-provider communication about CRCS as evidenced by the low percentage of respondents, without a provider advising a specific CRCS test, who correctly reported having any CRCS test when they had it (sensitivity) and low percentage of respondents, whose provider advised a specific CRCS test, that correctly reported not having any CRCS test and did not (specificity).

Strengths of our study include the use of a standardized questionnaire to ascertain self-reported screening, inclusion of covariates that have not been examined previously, a diverse sample with a large number of African Americans, a relatively stable patient population where almost all endoscopies were done on site, and follow-up with outside providers to verify self-reports in the few cases where CRCS was done elsewhere. Nevertheless, the generalizability of our findings is limited because our sample consisted of insured men and women receiving care primarily from one practice.

To our knowledge, this is the most comprehensive assessment to date of the accuracy of self-reported measures of CRCS by respondent characteristics and healthcare utilization factors. Our findings show that self-report of any CRCS test within guidelines, as measured by concordance, sensitivity, and report-to-records ratio, is reasonably accurate across subgroups. However, more research is needed to replicate these findings in diverse settings and populations to better understand subgroup differences and improve measures of CRC.