Validation of a Questionnaire to Assess Self-Reported Colorectal Cancer Screening Status Using Face-to-Face Administration

Abstract

Purpose

To assess the accuracy of an NCI-developed colorectal cancer screening questionnaire.

Methods

We conducted 36 cognitive interviews and made iterative changes to the questionnaire to improve comprehension. The revised questionnaire was administered face-to-face to 201 participants. The primary outcome was agreement between questionnaire responses and medical records for whether or not a participant was up-to-date for any colorectal cancer screening test.

Results

Comprehension of descriptions and questions was generally good; however, the barium enema description required several revisions. The sensitivity of the questionnaire for up-to-date screening status was 94%, specificity was 63%, and concordance was 88%.

Conclusions

The modified questionnaire was highly sensitive for determining if a person was up-to date for any colorectal cancer screening test, although the specificity was low. Given the difficulty of obtaining all relevant records, self-report using this questionnaire is a reasonable option for identifying people who have undergone testing.

INTRODUCTION

Although medical records may be considered the standard for determining colorectal cancer (CRC) screening status, the records from different medical facilities are rarely linked or centralized. Furthermore, obtaining records from multiple sources is time-consuming in the clinical setting and particularly challenging in the research setting. Patient self-report is a potentially more accessible and less expensive data source, but the accuracy of self-report has not been extensively assessed, particularly for sigmoidoscopy, colonoscopy, and barium enema (BE) [1]. In addition, a standardized measure of CRC screening status could facilitate comparison across studies and potentially eliminate the need to validate a new measure for each study. Therefore, accurate and standardized measurement of self-reported CRC screening status and/or behavior should advance research in the investigation of temporal changes in screening utilization, evaluation of predictors of screening behavior, and assessment of the impact of interventions to improve screening rates [2].

The National Cancer Institute (NCI) developed a questionnaire to measure CRC screening status and behavior, and encouraged researchers to evaluate it for clinical and research use [2]. The questionnaire includes brief descriptions of fecal occult blood testing (FOBT), flexible sigmoidoscopy, colonoscopy, and BE, and 28 yes/no or multiple-choice questions to determine if an individual has undergone the tests and, if so, how often, when, and why.

While some prior studies have compared self-reports of CRC screening to other data sources, the CRC self-report surveys have varied considerably [3-5]. For instance, descriptions of screening tests differ among surveys, and misunderstanding of these descriptions may have contributed to inaccuracies in reported data. In addition, many surveys did not include all four CRC screening tests endorsed in published guidelines at the time of this study.

In this study, our aims were to: 1) evaluate comprehension of the NCI questionnaire and participants' recall of prior CRC screening tests using the NCI questionnaire and 2) validate the questionnaire against medical record documentation.

METHODS

The study was conducted in two parts and at two Veterans Affairs (VA) medical centers, Durham VA and Minneapolis VA. In the first part, conducted only at the Durham VA, we used cognitive interviews to evaluate comprehension of the NCI questionnaire and participants' recall of prior CRC screening tests using the NCI questionnaire. We then revised the questionnaire based on the outcomes of the cognitive interviews. In the second part, conducted at both sites, we used the revised questionnaire to validate self-reported screening status against medical record documentation. The study was approved by the Institutional Review Boards of the two sites. All participants provided written informed consent prior to participation. Participants received either a $20 (cognitive interview) or $10 (validation) incentive.

Part 1 - Cognitive Interviews

Recruitment

We conducted cognitive interviews with 36 veterans at the Durham site to evaluate comprehension of the questionnaire and recall of prior CRC testing. Inclusion criteria were being age 50 years or older and having at least one visit to a primary care clinic at the Durham Veterans Affairs (VA) Medical Center within the past 12 months. Patients with a diagnosis of dementia were excluded. The local facility's electronic medical record (VISTA) was used to generate a list of patients meeting inclusion and exclusion criteria who also had a scheduled appointment for any clinic at the Durham VA Medical Center within the following six months. Within two months of their scheduled appointment, patients were sent a letter of introduction that included a toll-free number to call if they did not wish to participate. Patients who did not respond to the letter were contacted by the research assistant to arrange a face-to-face meeting. A quota sample was then recruited to represent both sexes and African American and Caucasian patients. Six to 10 participants were recruited to evaluate the original questionnaire and each revision during a total of four rounds. The overall recruitment rate (number enrolled/number of recruitment telephone calls) was 36/50. Cognitive interviews were conducted between August and October 2005 and were audio recorded and transcribed.

Cognitive Interviews

We used retrospective verbal probing to conduct the cognitive interviews. This method is appropriate in later stages of questionnaire development because it simulates survey administration in the field [6]. The interviewer read aloud the screening test descriptions and corresponding survey items following the intended skip pattern. Afterward, the interviewer probed participants about the test descriptions and the questions they answered. Participants were asked to explain their understanding of key terms, paraphrase test descriptions and survey items, explain their responses to survey items, and indicate their level of confidence about those responses.

Analysis

Responses were sorted into categories: 1) unclear question wording, 2) variable interpretation of concepts, and 3) inadequate or unnecessary response categories. The research team discussed and addressed these issues, and when necessary, revised the wording of the questionnaire. The revised questionnaire was then used for the next cycle of cognitive interviews, with the process repeated until all identified problems with the questionnaire were resolved.

Part 2 - Validation study

Recruitment

In the second part of our study, we recruited patients from both the Durham and Minneapolis VA Medical Centers. These sites were chosen to include veterans from both rural and urban environments and thus represent geographic diversity. Eligibility criteria were age 50 years and older and current enrollment in a primary care clinic at either study site. The local facilities' electronic medical records (VISTA) were used to generate a list of patients meeting inclusion and exclusion criteria. We randomly selected potential participants at each site from this list, and we oversampled women for a goal of 25% of the sample. At the Durham site, we excluded people who participated in the cognitive interviews.

A letter of introduction was mailed that included a toll-free number for patients to call if they did not wish to participate. Patients who did not respond to the letter were contacted by the research assistant at each site to arrange a face-to-face meeting. Among eligible invited study candidates, the recruitment rate was 100/164 (61%) in Durham and 101/139 (73%) in Minneapolis. During the interview, the research assistant administered the revised NCI questionnaire (appendix) and a demographics questionnaire. All participants reporting tests at non-VA facilities were also asked to sign a release of information form and to provide facility and physician contact information to allow us to request non-VA records.

Medical Record Abstraction

We standardized the protocol for medical record abstraction of the CRC screening tests and other data and provided on-site training for research assistants at both sites.

For VA records, research assistants abstracted the electronic medical record, the endoscopy database and the laboratory database at each site, documenting dates and types of CRC screening tests as well as indications for the tests. Colonoscopy, sigmoidoscopy and BE data were abstracted for up to 10 years prior to the interview date and FOBT data were collected for up to 5 years prior to the interview date. In addition to the care received at the participating VA site, the VA electronic medical record included documentation of care received at any VA facility in the US.

Non-VA medical records were obtained by first contacting the facility by phone and then sending the signed release of information documents per that facility's protocol. We asked for any colonoscopy, sigmoidoscopy, or BE reports, notes, results for the prior 10 years and any laboratory results or notes from stool blood tests/FOBT/occult blood tests in the prior 5 years. Study personnel called the facility's medical records department until the requested records were sent. The same data were abstracted from the non-VA records as from the VA records.

All abstracted data were collected on standardized collection forms and then double entered into an electronic database. Discrepancies between the two entries were resolved using the hard copy form. This method was used to avoid typographical errors and improve data quality.

We had data from VA records on all patients. Enrollment in primary care was an inclusion criterion, but there was no minimal enrollment time. Thus, we had VA records for as long as a participant had received any care in the VA health system. We were able to obtain documentation from the non-VA facilities for all 48 participants who reported non-VA care at the Minneapolis site and for all but 3 of the 27 participants at the Durham site reporting at least one non-VA procedure.

Analysis of Validation Data

The primary outcome of the validation study was agreement between the NCI questionnaire and medical records as to whether patients were up-to-date for CRC screening. The primary measures of agreement were sensitivity and specificity of the NCI questionnaire using the combined VA and non-VA medical record as the referent standard. We chose to use up-to-date for any CRC test, rather than up-to-date for specific modalities, as the primary outcome because it is the most clinically relevant endpoint. It also is the endpoint for quality assessment of screening rates by VA performance measures and non-VA quality indicators such as HEDIS (Health Plan Employer Data and Information Set).

Up-to-date screening status from self-report as well as from the medical record was determined based on most current published guidelines at the time of the study [7], and was defined as having completed an FOBT within a year, a flexible sigmoidoscopy or a BE within 5 years, or a colonoscopy within 10 years [7]. The time intervals (rather than month/year) were used to determine up-to-date status from the questionnaire responses. If a participant reported having undergone a CRC test, but responded “don't know” to the time interval since the test, then the participant was classified as not-up-to-date for that CRC test. The month and year were used to determine up-to-date status from the medical record. If, for example, if a participant correctly reported a colonoscopy within the past 10 years but incorrectly reported flexible sigmoidoscopy in the past 5 years, then the person would be counted as up-to-date according to guidelines. Up-to-date status for individual modalities only considered that particular test. Those who did not fulfill the criteria for up-to-date were considered “not up-to-date”. To be classified as not up-to-date, a participant had to be not up-to-date by medical records for all of the four possible CRC screening tests. The definition of up-to-date will overestimate sensitivity while the definition of not-up-to-date will underestimate specificity.

We calculated point estimates for the sensitivity, specificity, concordance (overall agreement), positive predictive value, and negative predictive value of the screening questionnaire for overall screening status and for each of the four procedures. We used the modified Wald method to calculate 95% confidence intervals [8]. In addition, we calculated report-to-records ratios. The report-to-records ratio was the number of individuals who reported up-to-date screening status divided by the number of individuals who had an up-to-date screening test documented in the medical record. A report-to-records ratio greater than 1 indicates over reporting. Positive and negative predictive values are useful in the clinical setting because they estimate the probability that the self-report is accurate. All measures were calculated separately by site (Durham versus Minneapolis) and for both sites combined. For all measures, values ≥0.9 were considered “excellent” agreement and values ≥0.8 were considered “good” agreement, as previously categorized by Tisnado et al [9].

Sample Size Estimates for the Validation Portion of the Study

Sample size estimates for this study were based on desired precision of the sensitivity and specificity of up-to-date screening status. Using confidence intervals for proportions methods [10], we determined that a total sample size of approximately 200 (half at each site) would be sufficient to provide estimates of sensitivity and specificity in the 80%–90% range for CRC screening status with margins of error of 0.05–0.075.

RESULTS

Part 1 - Cognitive Interviews

Thirteen females (7 black, 6 white) and 23 males (10 black, 13 white) were interviewed during four iterative cycles. All but one participant had undergone at least one CRC screening test. Changes to the questionnaire were made after each interview cycle to address areas of confusion uncovered during the cognitive interviews.

Comprehension of descriptions and questions was good for FOBT, colonoscopy, and flexible sigmoidoscopy. Across all rounds, only 2 of 36 participants confused sigmoidoscopy with colonoscopy. The BE description, however, was not well understood. Although the term “barium” was associated with a GI test for many participants, the BE was easily confused with other gastrointestinal (GI) tests, particularly the upper GI series. The test description was revised and retested in three rounds of interviews until all participants understood the BE description (Table 1). In addition to uncovering comprehension problems, we found that 67% (24/36) of the respondents had difficulty recalling the date (month and/or year) for one or more of the tests, but were able to choose among specified time intervals. While these 24 participants were not confident about exact dates, they were confident about the timeframe. Difficulty remembering the exact test date appeared to be unrelated to the specific procedure type.

Table 1.

Iterative Revisions of Barium Enema Description

Round 1: Original wording from (2).	Barium enema, or a lower GI series, is another test to check for colon cancer. X-rays are taken of the colon after barium or barium and air are given by enema. The day before the test you are asked to drink a lot of liquids and to take laxatives. No solid food is permitted.
Round 2	Barium enema, or a lower GI series, is another test to check for colon cancer. X-rays are taken of the colon after barium or barium and air are given by enema into your rectum. The day before the test you are asked to drink a lot of liquids and to take laxatives. No solid food is permitted.
Round 3	Another test to check for colon cancer is called barium enema, or lower GI series. Barium or barium and air are put into your rectum. While you hold in the barium, an x-ray is taken of the colon. This is different than an upper GI series, where you drink the barium before having an x-ray taken of your stomach or small intestines. To prepare for the barium enema, or lower GI series, you are asked to drink a lot of liquids and take laxatives the day before the test. No solid food is permitted.
Round 4: Final revision.	Another test to check for colon cancer is called barium enema, which is also known as a lower GI series. Barium or barium and air are put into your rectum by enema. You do not drink the barium. While you hold the barium in your rectum, an x-ray is taken of the colon. To prepare for the barium enema, you are asked to drink a lot of liquids and take laxatives the day before the test. No solid food is permitted.

Part 2 – Validation Study

A total of 201 participants completed the questionnaires. With the exception of race (Durham subjects were more likely to be African-American), the Durham and Minneapolis samples were similar demographically (Table 2). By medical record documentation, approximately 80% at each site had undergone at least one CRC test within the recommended time interval, but participants at the Minneapolis site were more likely to be up-to-date for two or more CRC screening modalities. In addition, the pattern of tests differed somewhat between sites (Table 3). Sigmoidoscopy, either alone or in combination with other tests, was more common at the Minneapolis site (20% vs. 5%), although a similar proportion of participants at each site had undergone colonoscopy within the prior 10 years (53% Minneapolis, 51% Durham) (Table 3).

Table 2.

Validation Sample Characteristics

	Overall (N=201) Percent (n)	Durham (N=100) Percent (n)	Minneapolis (N=101) Percent (n)
Male	73.6% (148)	74% (74)	73.3% (74)
Age
Mean (SD)	66.5 (10.2)	64.2 (10.2)	68.7 (9.7)
Race
White	78.1% (157)	65.0% (65)	91.1% (92)
Black	16.4% (33)	26.0% (26)	6.9% (7)
Other	5.5 (11)	9.0 (9)	2.0 (2)
Highest Education Level
Elementary	2.0% (4)	2.0% (2)	2.0% (2)
Some High School	6.0% (12)	5.0% (5)	6.9% (7)
HS Grad/GED	28.9% (58)	31.0% (31)	26.7% (27)
Some College/Tech School	33.8% (68)	35.0% (35)	32.7% (33)
College Graduate	29.4% (59)	27.0% (27)	31.7% (32)
Overall Colorectal Cancer Screening Status*
Not Up-to-date	19.9% (40)	20.0% (20)	19.8% (20)
Up-to-date	80.1% (161)	80.0% (80)	80.2% (81)
Number of “up-to-date” CRC tests*
0	19.9% (40)	20.0% (20)	19.8% (20)
1	57.7% (116)	66.0% (66)	49.5% (50)
2	19.9% (40)	12.0% (12)	27.7% (28)
3	2.0% (4)	2.0% (2)	2.0% (2)
4	0.5% (1)	0.0% (0)	1.0% (1)

^*Indicates the number of CRC testing modalities (not the total number of individual tests) for which the subject was “up-to-date” based on medical record review. 0 indicates that the subject had not undergone any CRC test within recommended guideline timeframes.

Table 3.

Specific Patterns of CRC Testing Documented in Medical Records

	Overall (N=201)		Durham (N=100)		Minneapolis (N=101)
Colorectal Cancer Screening Test(s)	Frequency	Percent	Frequency	Percent	Frequency	Percent
No Test	40	19.9	20	20.0	20	19.8
FOBT Alone	33	16.4	26	26.0	7	6.9
Sigmoidoscopy	24	12.0	3	3.0	21	20.8
Alone	15	7.5	2	2.0	13	12.9
+ BE	1	0.5	0	0.0	1	1.0
+ FOBT	6	3.0	0	0.0	6	5.9
+ FOBT + BE	2	1.0	1	1.0	1	1.0
Colonoscopy	88	43.8	49	49.0	39	38.6
Alone	68	33.8	38	38.0	30	29.7
+ BE	3	1.5	2	2.0	1	1.0
+ FOBT	17	8.5	9	9.0	8	7.9
Colonoscopy +			2	2.0	14	13.9
Sigmoidoscopy	16	8.0
Jointly	13	6.5	1	1.0	12	11.9
+ BE	1	0.5	1	1.0	0	0.0
+ FOBT	1	0.5	0	0.0	1	1.0
+ FOBT + BE	1	0.5	0	0.0	1	1.0
Total Colonoscopy	104	51.8	51	51.0	53	52.5
Total Sigmoidoscopy	40	20	5	5.0	35	34.7

* Includes colonoscopies reported in both `Colonoscopy' and `Colonoscopy + Sigmoidoscopy' section

† Includes sigmoidoscopies reported in both `Sigmoidoscopy' and `Colonoscopy + Sigmoidoscopy' section

Almost all participants who reported having undergone a test also provided the time interval since that test; however, 2% of those reporting an FOBT and 3% of those reporting a colonoscopy answered “don't know” and were classified as not up-to-date for these tests as described in the methods section. Although the three participants with missing records from a non-VA facility were considered not to have undergone any CRC test at that facility, the participants each had documentation of other CRC testing (from the VA or another facility which did provide records), which defined them as “up-to-date”. For the full study sample (201 participants), the sensitivity for being up-to-date with CRC screening by at least one modality was 94% (Table 4). The specificity of the questionnaire to determine that no test had been undertaken within the recommended interval was 63%. The combined specificity (63%) was low relative to the specificities of the individual tests colonoscopies (86%), sigmoidoscopies (91%), FOBTs (84%) and BE (93%) (Table 4).

Table 4.

Questionnaire Test Characteristics ^*

Modality	Sensitivity† (95% CI)	Specificity† (95% CI)	Report to Records‡	Concordance† (95% CI)	Positive Predictive Value† (95% CI)	Negative Predictive Value† (95% CI)
CRC
Overall	0.94 (0.90–0.97)	0.63 (0.47–0.76)	1.04	0.88 (0.83–0.92)	0.91 (0.86–0.95)	0.74 (0.57–0.85)
Durham	0.95 (0.87–0.98)	0.60 (0.39–0.78)	1.05	0.88 (0.80–0.93)	0.90 (0.82–0.95)	0.75 (0.50–0.90)
Minneapolis	0.94 (0.86–0.98)	0.65 (0.43–0.82)	1.02	0.88 (0.80–0.93)	0.92 (0.83–0.96)	0.72 (0.49–0.88)
FOBT
Overall	0.75 (0.63–0.84)	0.84 (0.77–0.89)	1.13	0.81 (0.75–0.86)	0.66 (0.54–0.76)	0.89 (0.82–0.93)
Durham	0.78 (0.62–0.88)	0.84 (0.73–0.91)	1.06	0.82 (0.73–0.88)	0.74 (0.58–0.85)	0.87 (0.76–0.93)
Minneapolis	0.71 (0.51–0.85)	0.83 (0.73–0.90)	1.25	0.80 (0.71–0.87)	0.57 (0.39–0.73)	0.90 (0.81–0.95)
Sigmoidoscopy
Overall	0.75 (0.60–0.86)	0.91 (0.86–0.95)	1.10	0.88 (0.83–0.92)	0.68 (0.53–0.80)	0.94 (0.88–0.97)
Durham	0.80 (0.36–0.97)	0.93 (0.85–0.97)	2.20	0.92 (0.85–0.96)	0.36 (0.15–0.65)	0.99 (0.93–1.00)
Minneapolis	0.74 (0.58–0.86)	0.89 (0.79–0.95)	0.94	0.84 (0.76–0.90)	0.79 (0.62–0.90)	0.87 (0.76–0.93)
Colonoscopy
Overall	0.91 (0.84–0.96)	0.86 (0.77–0.91)	1.05	0.89 (0.83–0.92)	0.87 (0.79–0.92)	0.90 (0.82–0.95)
Durham	0.94 (0.83–0.99)	0.94 (0.83–0.98)	1.00	0.94 (0.87–0.97)	0.94 (0.83–0.99)	0.94 (0.83–0.98)
Minneapolis	0.89 (0.77–0.95)	0.77 (0.63–0.87)	1.09	0.83 (0.75–0.89)	0.81 (0.69–0.89)	0.86 (0.72–0.94)
BE
Overall	0.88 (0.51–1.00)	0.93 (0.88–0.96)	2.63	0.93 (0.88–0.95)	0.33 (0.17–0.55)	0.99 (0.97–1.00)
Durham	0.75 (0.29–0.96)	0.91 (0.83–0.95)	3.00	0.90 (0.82–0.95)	0.25 (0.09–0.54)	0.99 (0.93–1.00)
Minneapolis	1.00 (0.45–1.00)	0.95 (0.88–0.98)	2.25	0.95 (0.89–0.98)	0.44 (0.19–0.73)	1.00 (0.95–1.00)

^*calculated upper 95% confidence levels >1.00 were capped at 1.00

^†medical record documentation was the reference standard

^‡definition provided in text

Measures of test accuracy for the four individual tests were similar between study sites and were generally in the good to excellent range for sensitivity, specificity, and concordance (Table 4). However, the sensitivity was lower, ranging from 71%–78% for FOBT, 74%–80% for sigmoidoscopy, and 75%–100% for BE. The specificity of colonoscopy was 77% at Minneapolis compared to 94% at Durham.

The report-to-records ratios were approximately 1 (range 0.94–1.25) for all tests and sites with the following exceptions: sigmoidoscopy at Durham (2.20) and BE at both sites (overall 2.63, Durham 3.00, Minneapolis 2.25) (Table 4).

DISCUSSION

This study supports the sensitivity of the questionnaire to detect CRC screening among those who had actually been tested. Therefore, the questionnaire could be useful for avoiding duplicate testing by identifying patients who are already up-to-date for CRC screening. The questionnaire could also be used when measuring CRC screening quality indicators since it would also capture tests performed outside the specific health care system (in this case, the VA), which may not be easily assessed from available records. The questionnaire may also be useful in identifying individuals who have not undergone testing by the individual modalities. For example, with the trend in the US towards using colonoscopy for screening [11] and the recent VA directive (Directive 004–2007) to offer screening colonoscopy, it is notable that the questionnaire was very good at identifying not only subjects who were up-to-date, but also those who were not up-to-date with colonoscopy. Self-report on the questionnaire was 91% sensitive and 86% specific for screening status of colonoscopy, making the questionnaire highly valid for colonoscopy screening. Although the specificity for the other modalities was lower, this was a predictable result due to the a priori definitions of up-to-date and not up-to-date. In summary, sensitivity was improved at the expense of specificity.

In the first part of the study, the difficulty participants experienced with the BE description during the cognitive interviews was unexpected, but may be explained by the increased prevalence of GI fluoroscopy in VA settings compared to other medical settings. For example, a participant may have had an upper GI series and confused it with the lower GI series. Some confusion may have persisted even after the final revision leading to over reporting and hence a report-to-records ratio of 2.6. This over reporting is in contrast to the other modalities that had report-to-records ratios near 1.0, indicating neither over nor under reporting. As BE use for screening continues to decrease, the relevance of this finding for clinical or research applications will also diminish.

The frequent inability of participants to report procedure dates (month and year) may limit some research applications, but with only a few exceptions those who reported having one of the tests were able to choose an interval, such as “more than 1 year, but less than 5 years”. Although such intervals will allow researchers and clinicians to determine whether a test was performed within the guideline-recommended interval, this response pattern may be less useful for evaluating intervention outcomes [12]. Cognitive research has shown that participants may recall specific episodes for low-frequency events [13,14]. Therefore, probing about events surrounding a test may trigger recall, but could also increase the time burden of the questionnaire.

In addition to providing data to support the validity of self-reported CRC screening behavior, this study also offered a snapshot of “dual use” among VA primary care patients. Use of non-VA services by patients followed in the VA has likely been underestimated in the past. Almost half of all Minneapolis participants and over a quarter of all Durham participants reported having undergone at least one CRC screening test at a non-VA facility. The VA has a centralized electronic medical record that includes care received at any VA facility; however, this study points to potential limitations of that data source for some purposes. Although our non-VA record ascertainment was very successful, 96% overall, it required considerable resources that may not be available or sustainable in non-research clinical or quality assessment situations. Because investigating care received outside the VA system is important, self-report may be the most efficient means of obtaining that information.

As one of the first studies to validate the NCI questionnaire, this project could not answer all potential questions regarding the accuracy of the questionnaire, but was instead conceived as the next developmental step. For example, we powered the study to detect sensitivity and specificity of overall up-to-date status, the endpoint most commonly used for research and quality assessment. This approach did not provide as much power to detect sensitivity and specificity for individual screening modalities. Similarly, while we deliberately ensured a sample that included representation of women and minorities, we did not power the study to detect differences in questionnaire performance among all demographic subgroups. We chose two sites to increase generalizability, but these sites may not be representative of all VA facilities or other medical centers. It is notable that in spite of differences in local CRC testing practices and patient demographics, the performance characteristics of the questionnaire (Table 4) were very similar at each site.

Cognitive interviewing methodology has improved knowledge and behavior questionnaires related to cervical cancer [15] and colon cancer screening [2,16,17]. However, there is no easy solution to the quandary of how much to alter a questionnaire for improved accuracy in a given population and yet maintain enough of the original content and design for comparisons across studies. We attempted to minimally alter the questionnaire, while addressing major recall or comprehension problems. Each mode of administration (face-to-face, telephone, mail) has advantages and disadvantages. The justification for face-to-face interview in this study is: 1) this is the optimum method to collect process data, 2) the only testing of this instrument at the time we conceived the study was face-to-face; therefore, as the next step in development we continued in this mode prior to testing other modes, and 3) mailed surveys, while less expensive, may not be appropriate for a low literacy population [18].

The VA has a high CRC screening rate in general; it was 79% in this study (by medical record documentation). Therefore, this setting may limit the obtainable values for specificity. In addition, we only requested records for participants who reported that they had a CRC test performed outside the VA. We did not request contact information from all non-VA facilities ever visited (regardless of what care was received), nor did we request 10 years' worth of medical records from each of these institutions. While this approach would have perhaps offered a more thorough investigation of false and true negatives, it was impractical. Finally, we designed this study to estimate the accuracy of self-reported CRC screening status by any recommended modality. The precision of validity measures for less frequently used CRC tests, such as BE, was limited, but may also be of minimal clinical importance.

In conclusion, self-reported CRC screening behavior, as obtained by the modified NCI questionnaire, is highly sensitive in a diverse VA population. In addition, the questionnaire is highly sensitive and has good specificity for colonoscopy. Given the difficulty in obtaining charts and administrative data from multiple facilities, self-report using the NCI survey is an attractive option in practice and research.

Appendix Final Modified Colorectal Cancer Screening Questionnaire

Stool blood test or Fecal Occult Blood Test (FOBT)

Read bolded instructions to subject:

The following questions are about the stool blood test, also known as a fecal occult blood test, a test to check for colon cancer. It is done at home using a set of 3 cards to determine whether the stool contains blood. You smear a sample of your fecal matter or stool on a card from 3 separate bowel movements and return the cards to be tested.

I a) Before this test was described, had you ever heard of a fecal occult or stool blood test?

___Yes

___No (skip to Sigmoidoscopy and Colonoscopy section)

___Don't know (skip to Sigmoidoscopy and Colonoscopy section)

___Refused (skip to Sigmoidoscopy and Colonoscopy section)

___Missing (skip to Sigmoidoscopy and Colonoscopy section)

If “yes” to “ever heard” (1a):

b) Have you ever done a stool blood test using a “home” test kit?

___Yes

___No (skip to Sigmoidoscopy and Colonoscopy section)

___Don't know (skip to Sigmoidoscopy and Colonoscopy section)

___Refused (skip to Sigmoidoscopy and Colonoscopy section)

___Missing (skip to Sigmoidoscopy and Colonoscopy section)

If “yes” to “ever done”(1b):

c) How many home stool blood tests have you done in the last 5 years?

___Record response

___Don't know

___Refused

___Missing

d) When did you do your most recent home stool blood test?

______Record response

______Don't know (go to f)

______Refused (go to f)

______Missing (go to f)

If you did your most recent test within the past year:

e) What was the month and year?

________Month

_________Year

______Don't know

______Don't know

______Refused

______Refused

______Missing

______Missing

f) Why did you do your most recent home stool blood test?

______It was part of a routine exam or check-up

______It was because of a symptom or health problem

______It was follow-up of an earlier abnormal test

______Don't know

______Refused

______Missing

g) When did you do the home stool blood test before your most recent one?

______None before the most recent (skip to Sigmoidoscopy and Colonoscopy section)

______A year ago or less

______More than 1, but not more than 2 years before

______More than 2 years before

______Not sure/Don't know

______Refused

______Missing

h) Why did you have that home stool blood test

______It was part of a routine exam or check-up

______It was because of a symptom or health problem

______It was follow-up of an earlier abnormal test

______Don't know

______Refused

______Missing

Sigmoidoscopy and Colonoscopy

Read all bolded instructions to subject:

The following questions are about Sigmoidoscopy and Colonoscopy, two other tests to check for colon cancer. Both tests examine the colon using a narrow, lighted tube that is inserted in the rectum. Sigmoidoscopy only examines the lower part of the colon while Colonoscopy examines the entire colon.

With the Sigmoidoscopy:

• You are awake.

• You are able to drive yourself home.

• You are able to resume your normal activities.

With the Colonoscopy:

• You are given medicine through a needle in your arm to make you sleepy.

• You need someone to drive you home.

• You may need to take the rest of the day off from your usual activities.

II a) Before these tests were described, had you ever heard of SIGMOIDOSCOPY?

___Yes

___No (skip to Colonoscopy section)

___Don't know (skip to Colonoscopy section)

___Refused (skip to Colonoscopy section)

___Missing (skip to Colonoscopy section)

If “yes” to “ever heard” (2a):

b) Have you ever had a SIGMOIDOSCOPY?

___Yes

___No (skip to Colonoscopy section)

___Don't know (skip to Colonoscopy section)

___Refused (skip to Colonoscopy section)

___Missing (skip to Colonoscopy section)

If “yes” to “ever had”(2b):

c) How many SIGMOIDOSCOPY exams have you had in the last 10 years?

____Record response

____Don't know

____Refused

____Missing

d) When was your most recent SIGMOIDOSCOPY?

_____It was more than 5 but not more than 10 years ago

_____It was more than 10 years ago

_____Not sure/don't know (go to f)

_____Refused (go to f)

_____Missing (go to f)

e) What was the month and year of your most recent SIGMOIDOSCOPY?

________Month

_________Year

______Don't know

______Don't know

______Refused

______Refused

______Missing

______Missing

f) Why did you have your most recent SIGMOIDOSCOPY?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing

g) When did you have the SIGMOIDOSCOPY before your most recent one?

_____None before the most recent (skip to Colonoscopy)

_____It was more than 1 but not more than 5 years before

_____It was more than 5 years before

_____Don't know

_____Refused

_____Missing

h) Why did you have that SIGMOIDOSCOPY?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing

COLONOSCOPY

Read bolded instructions to subject:

Now I will ask you about colonoscopy, a test that uses a long tube and examines the entire colon, with the COLONOSCOPY:

• You are given medicine through a needle in your arm to make you sleepy.

• You need someone to drive you home.

• You may need to take the rest of the day off from your usual activities.

III a) Before these tests were described, had you ever heard of COLONOSCOPY?

___Yes

___No (skip to Barium Enema section)

___Don't know (skip to Barium Enema section)

___Refused (skip to Barium Enema section)

___Missing (skip to Barium Enema section)

If “yes” to “ever heard”(3a):

b) Have you ever had a COLONOSCOPY?

___Yes

___No (skip to Barium Enema section)

___Don't know (skip to Barium Enema section)

___Refused (skip to Barium Enema section)

___Missing (skip to Barium Enema section)

If “yes” to “ever had”(3b):

c) How many COLONOSCOPY exams have you had in the last 10 years?

___Record response

___Don't know

___Refused

___Missing

d) When was your most recent COLONOSCOPY?:

_____It was a year ago or less

_____It was more than 1 but not more than 5 years ago

_____It was more than 5 but not more than 10 years ago

_____It was more than 10 years ago

_____Don't know (go to f)

_____Refused (go to f)

_____Missing (go to f)

e) What was the month and year of your most recent COLONOSCOPY?

________Month

_________Year

______Don't know

______Don't know

______Refused

______Refused

______Missing

______Missing

f) Why did you have your most recent COLONOSCOPY?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing

g) When did you have the COLONOSCOPY before your most recent one?

_____Record response

_____None (skip to Barium enema section)

_____Don't know

_____Refused

_____Missing

h) Why did you have that COLONOSCOPY?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing

Barium Enema

Read bolded instructions to subject:

Another test to check for colon cancer is called barium enema, or lower GI series. Barium or barium and air are put into your rectum. While you hold in the barium, an x-ray is taken of the colon. This is different than an upper GI series, where you drink the barium before having an x-ray taken of your stomach or small intestines. To prepare for the barium enema, or lower GI series, you are asked to drink a lot of liquids and take laxatives the day before the test. No solid food is permitted.

IV a) Before this test was described, had you ever heard of barium enema or lower GI series?

___Yes

___No (end questionnaire, go to demographics)

___Don't know (end questionnaire, go to demographics)

___Refused (end questionnaire, go to demographics)

___Missing (end questionnaire, go to demographics)

If “yes” to “ever heard”(4a):

b) Have you ever had a barium enema?

___Yes

___No (end questionnaire, go to demographics)

___Don't know (end questionnaire, go to demographics)

___Refused (end questionnaire, go to demographics)

___Missing

If “yes” to “ever had”(4b):

c) How many barium enemas have you had in the last 10 years?

_____________Record response

_____________Don't know

_____________Refused

_____________Missing

d) When was your most recent exam?

_____It was a year ago or less.

_____It was more than 1 but not more than 5 years ago

_____It was more than 5 but not more than 10 years ago

_____It was more than 10 years ago

_____Don't know (go to f)

_____Refused (go to f)

_____Missing (go to f)

e) What was the month and year of your most recent barium enema?

________Month

_________Year

______Don't know

______Don't know

______Refused

______Refused

______Missing

______Missing

f) Why did you have your most recent barium enema?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing

g) When did you have the barium enema before your most recent one?

________Month

_________Year

______Don't know

______Don't know

______Refused

______Refused

______Missing

______Missing

h) Why did you have that barium enema?

_____It was part of a routine exam or check-up.

_____It was because of a symptom or health problem.

_____It was follow-up of an earlier abnormal test.

_____Don't know

_____Refused

_____Missing