Utilizing NCCN Practice Guidelines to Measure the Quality of Colorectal Cancer Care in the Veterans Health Administration

Abstract

Introduction

Clinical practice guidelines can be used to help develop measures of quality of cancer care. This paper describes the use of a Cancer Care Quality Measurement System (CCQMS) for monitoring these measures for colorectal cancer in the Veterans Health Administration.

Methods

The CCQMS assessed practice guideline concordance primarily based on colon (14 indicators) and rectal (11 indicators) cancer care guidelines of the National Comprehensive Cancer Network (NCCN). Indicators were developed with input from VHA stakeholders with the goal of examining the continuum of diagnosis, neoadjuvant therapy, surgery, adjuvant therapy, and survivorship surveillance and/or end-of-life care. In addition, 9 measures of timeliness of cancer care were developed. The measures/indicators formed the basis of a computerized data abstraction tool that produced reports on quality of care in real-time as data were entered.

Results

The tool was developed for a 28-facility learning collaborative, the Colorectal Cancer Care Collaborative (C4), aimed at improving CRC care quality. Data on 1,373 incident stage I-IV CRC cases were entered over approximately 18 months. Data were used to target and monitor quality improvement activities. The primary opportunity for improvement involved surveillance colonoscopy and services in patients after curative intent treatment.

Conclusions

NCCN guidelines were successfully used to develop a measurement system for VHA research-operations quality improvement partnership.

INTRODUCTION

The Veterans Health Administration (VHA) is the largest provider of cancer care in the United States. With approximately 5.5 million patients receiving care under the auspices of 153 medical centers in 2008, the VHA has the largest fully-integrated delivery system in the country.¹ The system treats approximately three percent of U.S. cancer cases (>43,000 in 2005). As in the rest of the country,² colorectal cancer (CRC) is the third most common cause of cancer in the VHA.³ There were >4,600 new cases of CRC entered into the Veterans Affairs (VA) Central Cancer Registry in 2005 (11% of VHA cancer cases).

Recognizing the need to reduce the time from positive screening to diagnosis and enhance the use of guideline-concordant care, the VHA began the Colorectal Cancer Care Collaborative (C4) quality improvement program in 2005. Between February 2007 and April 2008, C4 conducted the Colorectal Cancer Treatment Improvement Collaborative project. As part of this quality improvement effort, the Cancer Care Quality Measurement System (CCQMS) was developed to collect data on cancer care. CCQMS produces reports on the degree of guideline-concordant care based on the National Comprehensive Cancer Network (NCCN) practice guidelines as well as timeliness of CRC care.^{4, 5} The measurement tool was developed to evaluate the continuum of CRC care, from diagnosis through treatment and surveillance.

While the existence of this tool has been previously noted as part of summaries of the C4 initiative,^{4, 5} the purpose of this paper is to describe the development of specific quality indicators and CCQMS computerized tool and use of the CCQMS among the 28 facilities participating in the quality improvement collaborative.

METHODS

The development and use of the CCQMS was approved by the Institutional Review Board of the Durham Veterans Affairs (VA) Medical Center. The tool was used by VA medical centers that volunteered to participate in the 28-facility VHA Colorectal Cancer Treatment Improvement Collaborative.

Setting – VA Colorectal Cancer Treatment Improvement Collaborative

Based on the quality improvement collaborative methodology developed by the Institute of Healthcare Improvement in the 1990’s,⁶ the VHA conducted a collaborative focused on enhancing the guideline concordance of care, timeliness of care, and patient experience with care among Veterans with CRC. The collaborative has been fully described elsewhere.^{4, 5} Briefly, a core feature of collaboratives is utilizing rapid cycle quality improvement – Plan-Do-Study-Act cycles – to test improvements to the care system using data related to quality of care.^{7, 8} VHA Office of Quality and Performance (since renamed Office of Informatics and Analytics) invited each VHA regional Veterans Integrated Service Network (VISN) to nominate facilities for participation. Participating medical centers had to commit to both quality improvement and measurement activities in a participation contract signed by the medical center director, chief of staff, and chief nursing officer. Teams consisting of physicians, nurses, other clinicians, and non-clinician administrators planned and worked with facilities to conduct tests of change. In the C4 treatment collaborative, teams learned from each other through two in-person learning sessions, monthly conference calls, an email listserv, calls on special topics (e.g. ensuring surveillance colonoscopies for CRC survivors), and examples of improvement tools collaboratively developed with collaborative staff. Central to the collaborative was the use of data to target improvement activities and monitor changes. These data were collected using the CCQMS.^{4, 5}

Data Collection Tool – Cancer Care Quality Measurement System (CCQMS)

The CCQMS was developed with the intent of identifying facility-level and VHA-wide deviations from established standards of care to better enable cancer care quality improvement efforts. The data abstraction tool was written in C-Sharp (C#) programming language using the Microsoft.net framework and was comprised of approximately 230 data elements. Staff at the 28 facilities participating in the C4 treatment collaborative entered information pertaining to newly diagnosed colorectal cancer patients who received care at their VA medical center.

During the collaborative, the tool was housed within the VA-protected computer environment (i.e. behind the VA firewall). To assure that data security standards were met, all data were stored on a centralized VA server with no data stored on local computers or networks. When an abstractor answered an individual question, data were transferred to the centralized server. Additional features of the tool include: 1) a system for indicating whether case data require updating/completion; 2) the ability to search for entered cases; 3) a data verification feature; 4) helpful hints for specific questions; 5) printable reports updated in real time as data are entered; 6) ability to create reports by year of diagnosis; 7) links to key printable documents about the CCQMS (i.e. directions, question list, data dictionary, quality indicator definitions); and 8) a centralized e-mail helpdesk that operated during the collaborative.

CCQMS data were used to calculate 25 quality indicators based on the NCCN treatment guidelines for colon cancer⁹ and rectal cancer¹⁰ treatment. In addition, the CCQMS data abstraction tool calculated the number of elapsed days for time intervals between major components of cancer care, serving as a basis for nine timeliness of care measures.

Quality Indictors

Quality indicators were developed with the input of VHA constituencies such as the VHA Oncology Field Advisory Committee and teams participating in the C4 treatment improvement collaborative. They are based primarily on the guidelines of the NCCN. Founded in 1995, NCCN is consortium of 21 academic cancer centers that develops guidelines on a number of cancers.¹¹ Guidelines provide screening and treatment algorithms for each stage of cancer and are reviewed yearly. They represent a combination of evidence-based recommendations in areas for which peer-reviewed evidence is available and consensus-based recommendations for areas in which the evidence base is not fully developed.¹² The first guidelines for colon and rectal cancer screening and treatment were published in 1996.¹³ Except where explicitly stated, CCQMS quality measures described in this report were based on 2005 NCCN guidelines, giving users the opportunity to use the CCQMS to look at cases over multiple years. All NCCN-based quality indicators used in this study are considered to have some evidence-base and have uniform NCCN guideline-panel consensus (at least NCCN evidence category 2A).^{14, 15} Table 1 lists the quality indicators.

Table 1.

Quality Improvement Indicators for Patients with Incident Colorectal Cancer Developed for the Cancer Care Quality Measurement System (CCQMS)

Indicator Number	Indicator Description1	Indicator Denominator	Indicator Numerator	Indicator Rate
	Colon Cancer
C1	Complete Diagnostic Work-Up	Patients with invasive colon cancer [stages I–VI]	Patients receiving a complete workup, including total colonoscopy, complete blood count, platelets, chemistry profile, carcinoembryonic antigen (CEA) determination, chest radiography, and a computed tomographic (CT) scan of the abdomen and pelvis.2	52.1% (n = 984) Limitation: this measure requires that ALL workup components be completed; it does not account for clinical appropriateness nor patient refusal of specific workup components.
C2	Preoperative CEA Determination	Colon cancer patients with curative intent surgery [stages II–III]	Patients with preoperative CEA determination.	65.6% (n = 823)
C3	≥ 12 Lymph Nodes Examined	Colon cancer patients with resected tumors [stages II–III]	Patients with ≥12 lymph nodes examined by pathology.	68.3% (n = 681)
C4	Clear Surgical Margins	Colon cancer patients with resected tumors [stages II–III]	Patients with clear/negative margins (for surgeries completed 2006 or later)..	90.5% (n = 643)
C5	Referral to Medical Oncologist	Colon cancer patients [stages II–IV]	Patients with referral to a medical oncologist or documented reason for non-referral.	84.5% (n = 677)
C6	5-FU Adjuvant Chemotherapy	Colon cancer patients [stage III]	Patients referred to medical oncologist who then received 5-FU based therapy (intravenous or oral [capecitabine]) or documented reason why not.	100.0% (n = 222)
C7	Adjuvant Chemotherapy Within 8 weeks of surgery	Colon cancer patients [stage III]	Patients referred to medical oncologist who then received 5-FU based therapy (intravenous or oral [capecitabine]) with therapy started within 8 weeks of first positive biopsy or 8 weeks of surgery or documented reason for not receiving such therapy (for surgeries completed 2006 or later)..	65.3% (n = 144)
C8	5-FU Adjuvant Chemotherapy ≥ 6 months	Colon cancer patients [stage III]	Patients referred to medical oncologist who then received 5-FU based therapy (intravenous or oral [capecitabine]) for at least 6 months or documented reason for not receiving such therapy.	100% (n = 157)
C9	Follow-Up Colonoscopy – Obstructing Lesion	Colon cancer patients with preoperative obstructing lesion and living at least 3 months post surgery[stages II–III]	Patients who received a complete colonoscopy within 3–6 months of surgery or documented reason for not receiving procedure.3	11.3% (n = 106) Limitation: In addition to the low number of eligible cases, colonoscopies may have occurred outside the VHA or just outside the measure’s time window.
C10	Follow-Up Colonoscopy – Non-Obstructing Lesion	Colon cancer patients without preoperative obstructing lesion and with curative intent surgery[stages II–III]	Patients who received surveillance colonoscopy within 12 months of surgery or documented reason for not receiving procedure.4	22.6% (n = 328) Limitation: In addition to the low number of eligible cases, colonoscopies may have occurred outside the VHA or just outside the measure’s time window.
C11	Follow-Up History and Physical (H&P)	Colon cancer patients [stages II–III]	Patients with documented history & physical examination (H&P), and postoperative visits every 3 months for 2 years, then every 6 months in years 2–5 (total of 5 years).	59.8% (n = 823) Limitation: It is difficult to define a H&P visit. Abstractors were told an H&P does not necessarily have to be explicitly titled. Rather, a visit in which a provider evaluates the physical health of a patient and reviews their health history may constitute an H&P. Further, follow-up may have occurred outside the VHA or just outside the measure’s multiple time windows
C12	Follow-Up CEA Determination	Colon cancer patients [stages I–III]	Patients who received CEA blood tests after completion of curative treatment every 3–6 months for 2 years and every 6 months afterward for a total of 5 years.5	73.2% (n = 699) Limitation: Follow-up may have occurred outside of the VHA.
C13	Consultation with a Surgeon for Resectable Metastases	Colon cancer patients [stages II–III]	Patients who had a completed consultation with a surgeon or documented reason for non-referral.	85.7% (n = 203)
C14	Additional Therapy Offered for Metastatic Disease	Colon cancer patients [stage IV]	Patients offered additional therapy (surgery, chemotherapy, or hospice) or documented reason for not offering such treatment.	95.1% (n = 203)
	Rectal Cancer
R1	Complete Diagnostic Work-Up	Patients with invasive rectal cancer [stages I–IV]	Patients receiving complete workup, including complete blood count, platelets, chemistry profile, carcinoembryonic antigen (CEA) determination, endorectal/transrectal ultrasound or magnetic resonance imaging, chest radiography, a computed tomographic (CT) scan of the abdomen and pelvis, and referral to an enterostomal therapist for preoperative counseling if removal of the rectum is contemplated.2	39.3% (n = 389) Limitation: this measure requires that ALL workup components be completed; it does not account for clinical appropriateness nor patient refusal of specific workup components.
R2	Preoperative CEA Determination	Rectal cancer patients with curative intent surgery [stages II–III]	Patients with preoperative CEA determination.	71.3% (n = 268)
R3	Documented TME [patients diagnosed after 2005]	Rectal cancer patients with an anatomic bowel resection [stages II–III]	Patients with appropriate mesorectal excision (e.g. TME) (for surgeries completed 2006 or later).	20.5% (n = 156) Limitation: There is not uniform documentation for TME. TME may have occurred without use of that term in surgical notes.
R4	Clear Surgical Margins [patients diagnosed after 2005]	Rectal cancer patients with resected tumors [stages II–III]	Patients with clear/negative margins(for surgeries completed 2006 or later).	86.3% (n = 212)
R5	Referral to Radiation Oncologist	Rectal cancer patients [stages II–IV]	Patients with referral to a radiation oncologist or documented reason for non-referral.	81.1% (n = 265)
R6	Neo-adjuvant Chemo-Radiation Therapy	Rectal cancer patients who had curative intent surgery [stages II–III]	Patients referred for neo-adjuvant chemo and/or radiation or documented reason for not being referred.	57.0% (n = 265)
R7	5-FU Adjuvant Chemotherapy	Rectal cancer patients who had curative intent resection [stages II–III]	Patients received 5-FU based therapy (intravenous or oral [capecitabine]) adjuvant chemotherapy or documented reason for not receiving therapy.	25.7% (n = 265) Limitation: The low percentage may reflect debate over use of chemotherapy for stage II patients during the time of the collaborative.
R8	Follow-Up Colonoscopy – Obstructing Lesion	Rectal cancer patients with preoperative obstructing lesion [stages II–III]	Patients who received a complete colonoscopy within 3–6 months of surgery or documented reason for not receiving procedure.3	10.7% (n = 28) Limitation: In addition to low n, colonoscopies may have occurred outside the VA or just outside the time window.
R9	Follow-Up Colonoscopy – Non-Obstructing Lesion	Rectal cancer patients without preoperative obstructing lesion [stages II–III]	Patients who received surveillance colonoscopy within 12 months of surgery or documented reason for not receiving procedure.4	18.6% (n = 118) Limitation: In addition to low n, colonoscopies may have occurred outside the VA or just outside the time window.
R10	Follow-Up History and Physical (H&P)	Rectal cancer patients without preoperative obstructing lesion and curative intent surgery [stages II–III]	Patients with documented history & physical examination (H&P), and postoperative visits every 3 months for 2 years, then every 6 months in years 2–5 (total of 5 years).	64.7% (n = 190) Limitation: The strict definition of an H&P. Abstractors may not have included more general visits where CRC may have been addressed. Further, follow-up may have occurred outside of the VA.
R11	Follow-Up CEA Determination	Rectal cancer patients with curative intent surgery [stages I–III]	Patients who received CEA blood tests after completion of curative treatment every 3–6 months for 2 years and every 6 months afterward for a total of 5 years.5	72.8% (n = 243) Limitation: Follow-up may have occurred outside of the VA.

Notes

¹Except where noted otherwise, quality indicators are based on the 2005 National Comprehensive Cancer Network (NCCN) colon cancer (version 2) and rectal cancer (version 2) clinical practice guidelines. The exception is that C5, C8, R3 are based on information from the National Initiative on Cancer Care Quality (NICCQ) of the American Society of Clinical Oncology. In addition, C12 and R10 are based on the 2007 NCCN guidelines.

²The complete work-up measure only considers those procedures available at the location at which the primary work-up was completed. If there is a documented reason for not providing the recommended care, the indicator will be considered to be fulfilled. Additionally, endorectal ultrasound is not required for metastatic patients.

³Denominator includes patients who are at least 6 months post-operation.

⁴Denominator includes patients who are at least 12 months post-operation and alive.

⁵Beginning in 2007, NCCN guidelines suggest CEA tests every 3–6 months in the first two years postoperatively. The reporting algorithm allows for the most liberal interpretation of this guideline. Care of a patient receiving 1 CEA per year post-operatively will be considered to be concordant with the guideline. Abstraction is beginning with cases diagnosed in 2006 and 2007. As such, five-year data do not exist. To overcome this limitation, cases will be considered compliant if 2 or more H&Ps/CEAs have been performed within the first year post-surgery/post-curative treatment. Similarly, if 4 or more H&Ps/CEAs have been performed within the first two years post-surgery/post-curative treatment they will be considered compliant. Cases who did not undergo surgery or who died within the first year following surgery are excluded from the denominator. This algorithm will be reexamined as additional data becomes available.

⁶The specific measures described in this table are not part of the official VA quality monitoring systems.

⁷A limitation of all measures in this list is that care occurring outside of the VA healthcare system may not have been fully captured as part of quality improvement oriented data abstraction.

In addition, the CCQMS data abstraction tool calculated the number of elapsed days for nine time intervals (e.g. number of days between a patient’s colorectal cancer diagnosis and the first treatment date). While there is limited current evidence that timeliness of care impacts outcomes of CRC treatment,¹⁶ there is a general consensus among oncology societies that timely cancer care is an important indicator of cancer care quality and patient-centeredness and timely access to colorectal cancer care has been positively associated with patient satisfaction.^{17, 18} In addition, increasing the efficiency of care is a significant focus of the VHA.¹⁹ Table 2 lists specific timeliness measures. It should be noted that the indicators and measures detailed here were intended for use in improving quality of care and are not currently part of an ongoing national VHA quality measurement program.

Table 2.

Timeliness for Patients with Incident Colorectal Cancer Developed for the Cancer Care Quality Measurement System (CCQMS)

Indicator Number	Indicator Description1	Timeliness Measure Description	Median Days Between Major Treatment Events
T1	Time from colonoscopy to diagnosis.	Elapsed days between a colonoscopy and collection of a pathologic specimen which is diagnostic for colorectal cancer. Note that for cases diagnosed via colonoscopy, there may be zero elapsed days.	41.0 days (n = 263)
T2	Time from diagnosis to the first treatment date.	Elapsed days between collection of a pathologic specimen which is diagnostic for colorectal cancer and the first treatment date. The first treatment date could be the date of surgery, starting date of chemotherapy, or starting date of radiation therapy – whichever occurs first.	29.0 days (n = 1,162) Limitation: Diagnosis date could have occurred after the first treatment date if surgery occurred before pathologic diagnosis. Such a number would have been negative in the calculation.
T3	Time from diagnosis to informing patient about diagnosis.	Elapsed days between collection of a pathologic specimen which is diagnostic for colorectal cancer and the date the patient is told that they have cancer.	4.0 days (n = 1,005)
T4	Time from end of treatment to follow-up colonoscopy.	Elapsed days between the end of a patient’s treatment and the follow-up colonoscopy. The end of a patient’s treatment could be the date of surgery, ending date of chemotherapy (neoadjuvant or adjuvant), or ending date of radiation therapy – whichever occurs last.	288.0 days (n = 339) Limitation: Follow-up may have occurred outside of the VA.
T5	Time from completion of neoadjuvant radiation treatment to surgery.	Elapsed days between the ending date of neoadjuvant radiation treatment and the date of surgery.	53.0 days (n = 123)
T6	Time from completion of neoadjuvant chemotherapy to surgery.	Elapsed days between the ending date of neoadjuvant chemotherapy treatment and the date of surgery.	53.0 days (n = 108)
T7	Time from surgery to start of adjuvant chemotherapy.	Elapsed days between the ending the date of surgery and the beginning date of adjuvant chemotherapy treatment.	54.0 days (n = 308)
T8	Time from start date to end date of adjuvant chemotherapy.	Elapsed days between the beginning and the ending date of adjuvant chemotherapy treatment.	165.5 days (n = 220)
T9	Time from end of treatment to death (stage IV only).	Elapsed days between the end of a patient’s treatment and the date of death. The end of a patient’s treatment could be the date of surgery, ending date of chemotherapy (neoadjuvant or adjuvant), or ending date of radiation therapy – whichever occurs last.	100.0 (n = 62)

Notes

¹The C4 Timeliness Indicators were devised by clinical experts in the fields of oncology and gastroenterology with further input from surgery. While it may be assumed that shorter time periods are better, there is very little empirical evidence-base for determining optimal timing for cancer treatment. (All measures required that patients have incident, or newly diagnosed cancer.)

²Shorter periods of time, smaller number of elapsed days, are the goal of most C4 Phase II Timeliness Indicators. An exception is T8 in which a 6 month interval is suggested. Patients will only be included in the timeliness measures when data have been entered for both time points.

³The specific measures described in this table are not part of the official VA quality monitoring systems.

⁴A limitation of all measures in this list is that care occurring outside of the VA healthcare system may not have been fully captured as part of quality improvement oriented data abstraction.

Patient Inclusion Criteria

Inclusion in the CCQMS suggested to facilities participating in the collaborative was based on the following criteria: 1) histological or cytological confirmed diagnosis of colon or rectal cancer; 2) invasive cancer (in-situ diagnoses excluded); 3) primary tumor of the colon or rectum (metastases to colorectal sites from other primaries were excluded); 4) no prior invasive cancer of the colon or rectum (recurrences excluded); and 5) ≥ 18 years of age at diagnosis.

Facilities were instructed to exclude patients with a diagnosis of another type of cancer within two months of the diagnosis of colorectal cancer. Although the Surveillance Epidemiology and End Results [SEER] registry system defines simultaneous diagnoses as two separate primary tumors, of the same or different sites, diagnosed within a two-month period, there was no recommendation to exclude patients with more than one primary tumor of the same site (that is, two colon primaries) within two months.

Seventeen of the 27 facilities utilizing the CCQMS during the collaborative (1 facility entered only a very limited number of cases) also had local VA Central Cancer Registry processes for identifying patients with colorectal cancer. VA Central Cancer Registry case finding methods adhere to the standards established by the American College of Surgeons’ Commission on Cancer Facility Oncology Registry Data Standards (FORDS) Manual for data collection and definitions (the definitions can be found in the FORDS manual available at http://www.facs.org/cancer/coc/fordsmanual.html, accessed April 25, 2011).²⁰ Other facilities utilized clinic lists to identify patients.

Collaborative Process Quality Improvement Survey

As part of the quality improvement process for the collaborative, C4 organizers surveyed facility abstractors and other collaborative participants at the conclusion of the collaborative. That survey asked individuals who collected data for the CCQMS whether they disagreed or agreed with a series of statements about the tool. The mean of all answers from a given facility was taken as a facility-level score. Means and standard deviations were calculated for these facility-level scores. The 40 responding abstractors represented 22 of the 27 CCCQMS-utilizing facilities (81%)trhat utilized the CCQMS. Based on the responses from non-abstractor collaborative team members, information on plans to continue tracking quality indicators is available for an additional 3 facilitates for a total of 25 facilities.

Reporting of Quality Indicator Results

We report the percentage of patients who had guideline-concordant care as assessed (or measured) by the quality indicators and the mean number of days between major CRC treatment events. To be included in the denominator for a guideline concordance indicator or in results for a timeliness measure, patients had to meet inclusion criteria for that measure (see Tables 1–2). Unlike the reports generated for the facilities in the C4 collaborative, which required five or more eligible patients from a facility to have those patients included in the report, data from all eligible patients were included in analyses for the present paper. Further, results included in this reprot are based on a post-hoc analysis of data elements that were converted into a SAS data set and cleaned for inconsistent data using SAS version 9.2 (SAS Institute Inc., Cary, NC).

RESULTS

Use of the CCQMS

Between March 1, 2007 and May 15, 2008 data on approximately 1,373 patients with stage I-IV CRC (984 colon; 389 rectal) from 28 VA medical centers diagnosed primarily in 2006–2007 were entered into the C4 Cancer Care Quality Measurement System. The 28 VA facilities participating in the treatment improvement collaborative included at least one participating medical center from each of the 21 VHA regions. However, one of the 28 C4 facilities did not fully utilize the CCQMS [although data on a limited number of patients were entered]. The remaining 27 VA medical centers entered information on enough cases to utilize the reports feature of the CCQMS (at least five patients meeting inclusion criteria for one or more quality indicators). Men represented 98% of entered cases and the mean age was 69 years (standard deviation (SD) = 10.6). Race/ethnicity included 81% white, non-Hispanic, 15% black, non-Hispanic, 3% white, Hispanic, and 1% other race/ethnicity.

In addition to geographic variation, the 27 participating facilities that utilized the CCQMS included different degrees of complexity (VA measure based on types of services offered, organizational units, and size) [20 high complexity, 4 medium complexity, 3 low complexity],²¹ volume of CRC patients [6 facilities < 25 total cases each; 7 facilities = 25–50 total cases each; 7 facilities = 50–70 total cases each; 7 facilities = 71–110 total cases each], and cancer care accreditation [13 facilities, or approximately 48%, of participating facilities were accredited by the American College of Surgeons Commission on Cancer in 2005; an additional eight facilities (30%) were in the process of applying for accreditation].²¹ Characteristics of the 27 facilities utilizing CCQMS reports are detailed in Table 3.

Table 3.

Organizational Characteristics of VA Medical Centers Utilizing the CCQMS as part of the VA Colorectal Cancer Care Treatment Collaborative, n = 27^*

Characteristics of Oncology Program	Number
TOTAL ELIGIBLE PATIENTS IN CCQMS
< 25 patients	6
25 to 50 patients	7
50 to 70 patients	7
≥ 70 patients	7
HOSPITAL COMPLEXITY
High	20
Medium	4
Low	3
CANCER CARE ACCREDITATION
Accredited by the American College of Surgeons Commission on Cancer	13
CANCER SERVICES
Surgery	26
Medical Oncology	26
Radiation Oncology	13

^*Table does not report characteristics for the one facility participating in the quality improvement collaborative that did not fully utilize the CCQMS.

Facility cancer registrars conducted data abstraction at 13 of the 27 utilizing sites. Additionally, cancer registrars were integrally involved with CCQMS at an additional three sites. Nurses conducted data abstraction at six sites. The role of the CCQMS abstractor at two of the facilities was unknown.

The survey of quality improvement team members from the participating facilities indicated that that at the end of the collaborative, 17 of 25 reporting sites (68%) hoped to continue to track half or more than half of the CCQMS measures. Only two of the reporting facilities indicated that they did not intend to track any of the CCQMS measures.

It appears that the CCQMS was generally well-received. Based on 22 reporting facilities, the mean facility-level score on a one (strongly disagree) to five (strongly agree) scale for surveyed CCQMS data abstractors indicated good agreement with the following statements: 1) I understand what needs to be entered into each data entry field [mean = 4.44 (SD = 0.51)]; 2) I know where in the medical record to find the information for each data entry field [mean = 4.40 (SD = 0.67)]; 3) The information provided by the tool makes the time spent entering data worthwhile [mean = 3.88 (SD = 1.01)] and 4) The data reports are easy to understand [mean = 3.71 (SD = 1.05)].

The C4 teams received significant technical support and tool alterations in response to team feedback. Agreement scores for related survey items included: 1) I received the information/support that I needed from the Durham Data Abstraction Group [mean = 4.38 (SD = 0.79)]; 2) The on-line instructions and supporting information are very clear [mean = 3.95 (SD = 0.94)]; and 3) The tool has substantially improved since the first time I used it [mean = 4.39 (SD = 0.68)]. There was, however, still room to improve the tool. Related agreement scores included: 1) It is easy to enter data into the tool [mean = 3.42 (SD = 1.14)] and 2) The tool is as good as it can get [mean = 2.70 (SD = 0.88)].

Quality of CRC Care

CCQMS chart abstraction data were collected for quality improvement and not research purposes. A primary goal was to use the measurement tool to teach facilities about individual NCCN guidelines. The greatest opportunities for improvement involved surveillance colonoscopy and other services following curative intent treatment. Consistent with other findings inside and outside the VHA,²² the proportion of patients receiving follow-up colonoscopy within one year of surgical resection was low. For example, only 22.6% of 328 stage II-III colon cancer patients without preoperative obstructing lesions and at least a year of follow-up at the time of data abstraction had a documented colonoscopy within one year of surgery. Among patients who had a follow-up colonoscopy, the median number of days between end of curative intent treatment and follow-up colonoscopy [stage I-III patients] was 288.0. Consistent with other studies, the proportion of patients receiving guideline-concordant care during the curative-intent treatment process was significantly higher.^{22, 23} The specific proportion of patients receiving concordant care and mean number of days between major treatment events can be found in Tables 1 and 2 respectively.

DISCUSSION

NCCN guidelines provided the basis for quality indicators that were integrated into a computerized data abstraction system (not part of the VA electronic health record) that was successfully used by more than 25 VA medical centers. Use of NCCN guidelines allowed us to create a system that could be used to examine quality across the spectrum of CRC care, from diagnosis through neoadjuvant therapy, surgery, adjuvant therapy, and surveillance or end-of-life care.

The tool had important features that made it useful for the facilities. The most significant was the ability to provide immediate feedback to a facility using instantly-generated reports. These reports provided the individual facility results and allowed for extensive comparison across participating facilities. The reports could be easily printed for distribution to leadership and clinicians.

In addition, the tool was enhanced with a number of features, many of which were suggested by abstractors. These included helpful hints for specific questions (e.g. where to find information in the electronic health record), the ability to search for specific patients included in a quality indicator or timeliness measure, an email help desk that was used during the C4 collaborative, and regularly scheduled abstractor conference calls during the collaborative. Responsiveness to the needs of facilities was critical in making the tool useful for quality improvement.

The majority of participating facilities utilized cancer registrars to collect data for the CCQMS. These individuals are employed as part of the system for contributing data to the VA Central Cancer Registry. As a result they have extensive training related to chart abstraction for monitoring patterns of cancer care. Other facilities utilized registered nurses with extensive related clinical knowledge for chart abstractions.

The CCQMS had important limitations, many of which are currently being addressed through other VHA projects. Because this was an abstraction tool, not a data extraction tool, significant time was required to enter data. Anecdotal reports indicated that it was not uncommon for abstractors to spend up to two hours abstracting a complex case. We hypothesize that this is the reason that abstractors reported that there was still an opportunity to improve the computerized tool at the end of the C4 collaborative. For example, the proportion of patients with guideline concordant surveillance may be underestimated because facilities did not always enter data not related to initial treatments into the CCQMS. Because of the burden of data abstraction, several projects are underway in the VHA to automate data extraction. These include use of natural language processing (NLP)²⁴ to extract information from progress notes (e.g. number of lymph nodes resected), development of templated oncology notes, and extraction of data from existing data sources such as the VA Central Cancer Registry.

The tool was used for quality improvement purposes for specific facilities. While these data can be used to generate hypotheses about VHA system-wide quality challenges, facilities may have focused data collection on specific measures.

An important goal of the CCQMS measures was to teach facilities about NCCN guidelines. As a result, we categorized measures by those addressing colon versus those addressing rectal cancer care. When this is combined with the stage-specific nature of many guidelines, the number of patients is fewest for less common situations (e.g. rectal cancer patients of a specific stage). In addition, we noted that including multiple processes under one indicator (e.g. patients with documented history & physical examination (H&P) and postoperative visits every three months for two years, then every six months in years 2–5) greatly reduced the percentage of patients who appeared to be getting guideline concordant care, but did not provide details of the specific processes completed. Data collection may have been further complicated if patients received surveillance outside of the VA healthcare system. However, we included these indictors as a way of encouraging potential quality improvement efforts around the issue of survivorship.

Another quality improvement limitation was lack of measures related to patient experience of care. To address this limitation, the VA, Department of Defense, and the National Cancer Institute have developed and piloted a survey of patient-reported experiences with CRC care. This includes both a mailed patient and caregiver survey aimed at augmenting clinical data with information on patient experience.^{25, 26}

NCCN guidelines provided the basis for efforts in the VHA that have evolved over time. Quality improvement activities began with manual data abstraction but are now focusing on development of templated notes and automated data extraction. The experience of the VHA provides an example for other large healthcare systems seeking to use quality measurements to inform initiatives for enhancing the quality of cancer care.