Measuring Quality of Care in the Treatment of Colorectal Cancer: The Moffitt Quality Practice Initiative

Abstract

Purpose

The Moffitt Quality Practice Initiative (MQPI) is a practice-based system of quality self-assessment, the ultimate goal of which is to improve the quality of cancer care at a statewide level. The initial phase of this project focused on developing procedures, determining feasibility, and evaluating utility for assessing quality of care for colorectal cancer within an existing affiliate network.

Patients and Methods

Representatives from four oncology groups selected quality measures consistent with evidence-, consensus-, and safety-based guidelines that could be abstracted from medical records. Trained abstractors then reviewed records of all eligible colorectal patients seen by each practice in 2004. Frequencies of responses for each indicator were tabulated for overall and practice-specific level of adherence and were compared among practices.

Results

Adherence was uniformly high for several indicators, including confirmatory pathology report, staging information, and chemotherapy discussion or recommendation. Lower adherence was evident across practices for performance of carcinoembryonic tests and complete colonoscopic evaluations. Significant variation among practices was evident only for consent for chemotherapy.

Conclusion

The initial phase of MQPI demonstrated the feasibility and utility of assessing quality indicators for colorectal cancer among members of an existing affiliate network. Findings identified areas where adherence to care was uniformly high, but also identified areas where both overall and practice-specific adherence were less than optimal. These efforts lay the groundwork for expanding MQPI in several directions that have in common the potential to improve the quality of cancer care on a statewide basis.

Introduction

Since 1999, the Moffitt Cancer Center & Research Institute, a National Cancer Institute–designated Comprehensive Cancer Center, has been developing affiliations with strategically located hospitals and practices throughout Florida, the state with the second highest death rate from cancer in the United States.¹ Throughout the last 7 years, the Moffitt Affiliate Network has grown to include 17 community hospitals and more than 280 affiliated oncologists. It is estimated that upwards of 20% of all cancer patients in Florida are currently treated by network members. Affiliation activities include collaborations in the areas of cancer prevention and screening, continuing medical education, clinical service delivery, and clinical research. This infrastructure also provides a unique opportunity for the Moffitt Cancer Center to study and achieve one of its prime objectives—the establishment of higher standards of cancer care statewide.

In 1999, the Institute of Medicine's National Cancer Policy Board published “Ensuring Quality Cancer Care,” a landmark report that called attention to problems with the quality of cancer care in the United States.² The board concluded that for many people in the United States, there is a wide gap between the ideal versus the reality of their experience with cancer care. Among the recommendations was a call for the development of systems to measure and monitor quality of care using a core set of indicators.²

Stimulated by the board's report, a number of quality-of-care initiatives have been implemented by national organizations. A major example of such an effort is the National Initiative for Cancer Care Quality (NICCQ), which is sponsored by ASCO.³ Based on literature review and expert reviews, this project developed 36 quality measures for breast cancer and 25 quality measures for colorectal cancer.⁴ In the initial evaluation of these measures, data were collected from five metropolitan areas in the United States for patients newly diagnosed in 1998 with stage I to III breast cancer and stage II to III colorectal cancer.⁴ Findings indicated that among colorectal cancer patients, adherence ranged from 57% to 93% for individual indicators, and was less than 85% for 14 of the 25 indicators.

Although these findings identify important opportunities for improvement, they are based on geographically defined populations and, consequently, it is difficult to determine the care prescribed by individual institutions or practices. Toward this end, the Quality Oncology Practice Initiative (QOPI)⁵ was initiated with support from ASCO to provide oncology practices with a tool for self-examination that can promote excellence in cancer care. The QOPI process, based on retrospective review of medical records, has the capability of yielding measurements of practice quality that allow for comparisons across practices and over time. In the initial phase of this project, 11 quality indicators were evaluated in seven oncology groups in two rounds conducted 6 months apart. Quality indicators included issues related to patient safety (eg, use of chemotherapy flow sheets), application of evidence-based treatment (eg, use of serotonin antagonist with high emetogenic chemotherapy), and patient-centric care (eg, consent for chemotherapy treatment). Findings indicated significant variation in adherence among the practices on eight (73%) of the 11 indicators.

Building on both the NICCQ and QOPI efforts, the Moffitt Quality Practice Initiative (MQPI) was established with the ultimate goal of improving the quality of cancer care among all member institutions and throughout the state of Florida. The current report focuses on the methodology and findings of our initial experience conducting quality-of-care assessments at four of 17 member institutions. Similar to QOPI, the current project relied on retrospective reviews of medical records seen in selected medical oncology practices. In contrast to QOPI, the current project focused on two specific cancers (colon and rectal cancer) and included all cases seen in a calendar year. The project had three aims: demonstrate the feasibility of conducting a multisite evaluation of quality of care for colorectal cancer within an existing affiliate network; obtain practice-specific and aggregate information about adherence to quality care indicators; and provide a specific target or targets for quality improvement efforts.

Methods

Selection of Practices

At the time this project was conducted, the Moffitt Affiliate Network comprised 17 affiliate institutions located in the state of Florida. Selection of practices for participation was guided by the following considerations: (1) practice included more than one medical oncologist; (2) ability and agreement to provide experienced medical record abstractor; (3) agreement to participate in the development of the quality standards, data collection form, and data collection process; and (4) ability to contribute at least 40 cases of colon or rectal cancer seen in calendar year 2004. Three Moffitt Affiliate Network members were selected that met these criteria: Space Coast Medical Associates, Titusville, Florida; Center for Cancer Care and Research, Lakeland, Florida; and Sarasota Memorial Hospital, Sarasota, Florida. All three of these community-based oncology practices agreed to participate. Data were also collected at the Moffitt Cancer Center, a National Cancer Institute–designated comprehensive cancer center located at the University of South Florida. The project received approval from institutional review boards at each participating institution. To maintain patient privacy, collection of demographic data was limited to patient sex, and all records were coded with a unique project identifier before transmission to the central data collection site. Based on exempt approval status, written informed consent from patients was not required to access medical records.

Selection of Quality Measures

Selection of quality measures focused on assessing indicators consistent with evidence-, consensus-, and safety-based guidelines. Item selection began by reviewing those used previously by QOPI to develop a practice-based system of quality assessment.⁵ Additional items were constructed based on review of relevant literature and suggestions by project participants. An initial list of quality-of-care indicators was developed and approved by consensus during a meeting attended by representatives from all participating practices. The measures were further reviewed to ensure that they could be easily abstracted from office-based medical oncology records and that responses could be coded in binary terms (yes or no). The final list of indicators appears in Table 1. In addition to these quality indicators, an item was included for exploratory purposes to identify the number of cases diagnosed via routine screening.

Table 1.

Quality Indicators

Parameter	Indicator	Source
Patient safety: diagnostic ascertainment	Was “there a copy of the surgical pathology report confirming malignancy in the medical oncology office chart?”	QOPI5 and panel consensus
Patient safety: diagnostic ascertainment	Was “there an explicit statement of the patient's staging according to the AJCC or Dukes systems?”	Panel consensus
Application of evidence-based surveillance: use of CEA test	Was “there a blood test for CEA at least once prior to surgery or chemotherapy?'	ASCO guidelines,19 NCCN guidelines, and panel consensus
Application of evidence-based surveillance: use of CEA test	Was “there a blood test for CEA at least once in the 6 months after surgery or chemotherapy?”	ASCO guidelines,19 NCCN guidelines,14,20 and panel consensus
Patient safety: diagnostic ascertainment	For patients who did not have obstructive lesions, was “a complete evaluation of the colon performed within 12 months of diagnosis?”	NCCN guidelines14,20 and panel consensus
Application of evidence- and consensus-based treatment: use of chemotherapy	For patients with metastatic colon or rectal disease or nonmetastatic colon or rectal disease with lymph node involvement or rectal disease penetrating the rectal wall, “did the physician discuss, recommend, or refer for chemotherapy?”	NCCN guidelines, NICCQ,4 and panel consensus
Patient centric care: consent for chemotherapy treatment	For patients who received chemotherapy, was “there a signed consent for treatment in the chart or a practitioner's note that treatment was discussed and patient consented to treatment?”	QOPI5 and panel consensus
Patient safety: use of flow sheets	For patients who received chemotherapy, was “there a flow sheet with chemotherapy notes and blood counts?”	QOPI5 and panel consensus

Abbreviations: QOPI, Quality Oncology Practice Initiative; AJCC, American Joint Committee on Cancer; CEA, carcinoembryonic antigen; NCCN, National Comprehensive Cancer Network; NICCQ, National Initiative for Cancer Care Quality.

Medical Record Selection

Medical record reviews were conducted for all patients diagnosed with colon or rectal cancer in 2004 and seen by a medical oncologist at Moffitt Cancer Center and the three affiliate institutions. Patients younger than 18 years and those with anal carcinoma and synchronous colorectal malignancies were excluded. Other nonadenocarcinoma malignancies (eg, sarcomas, melanomas, lymphomas, neuroendocrine tumors, and gastrointestinal stromal tumors) were similarly excluded. For the purposes of this study, rectosigmoid cancer was classified as colon cancer.

Medical Record Review and Data Entry

A training manual for data identification, abstraction, and entry was developed and reviewed at a meeting of participating clinicians and data abstractors to ensure consistency across practices. A research physician from Moffitt Cancer Center was designated and trained as the chief abstractor for the project and trained and monitored all the other data abstractors. The training and monitoring was comprised of three phases. The first phase consisted of detailed on-site training. The chief abstractor reviewed five cases of colorectal cancer from 2003 (the year before the study year) with each abstractor to ensure accuracy and reliability of data collection. During the second phase, each abstractor reviewed five additional cases from 2003. The same medical records were reviewed independently by the chief abstractor and assessed for concordance. Additional training was provided if necessary before practices were approved for project initiation. The third phase occurred after the completion of the initial 15 cases at each practice; the chief abstractor reviewed three randomly selected cases to ensure ongoing quality of data collection and entry. This procedure was repeated following an additional second set of 15 cases from each practice. Abstractors were trained in data entry using a secured Web-based application designed for collection and transmission of data to the Moffitt Cancer Center. This application gave users the capability to skip questions that did not apply, thus minimizing possible data entry errors.

Statistical Analysis

Overall and practice-specific adherence rates were calculated for each indicator. Variation in adherence across practices was evaluated by conducting Fisher's exact tests with a two-tailed significance level of P < .05. An a priori statistical power evaluation was not conducted due to the exploratory nature of the project.

Results

Medical records for 260 patients were accrued and abstracted. The number of records per practice ranged from 46 to 102 (mean, 65 records). Fifty-eight percent of patients (n = 150) were male. Seventy-four percent of patients (n = 192) had been diagnosed with colon cancer. Of these patients, 64% had nonmetastatic disease (n = 122) and 36% had metastatic disease (n = 70). The remaining 26% of patients (n = 68) had been diagnosed with rectal cancer. Of these patients, 65% (n = 44) had been diagnosed with nonmetastatic disease, and 35% (n = 24) had been diagnosed with metastatic disease.

The percentage of patients whose cancer had been detected by routine screening ranged from 10% to 20% across practices (mean, 13%). Variation across practices was not statistically significant (P = .42).

Information about overall and practice-specific performance on quality indicators appears in Table 2. The rates for each practice are presented in masked form to preserve anonymity per prior agreement with participating institutions. Adherence rates were uniformly high (≥ 85%) and did not differ significantly (P values ≥ .10) by practice for the following indicators: pathology report confirming malignancy, staging according to American Joint Committee on Cancer or Dukes criteria, chemotherapy discussed or recommended, and chemotherapy flow sheet used. Adherence rates were slightly lower (75% to 89%) for performance of a carcinoembryonic antigen (CEA) test in the 6 months after surgery or chemotherapy, but did not vary significantly by practice (P = .28). Adherence rates were generally moderate (range, 37% to 67%) and did not differ significantly by practice (P values ≥ .23) for performance of a complete colon evaluation within 12 months of diagnosis and performance of a CEA test before surgery or chemotherapy. Finally, adherence rates did vary significantly by practice (P < .001) for consent for chemotherapy. While three practices demonstrated high rates of adherence to this indicator (≥ 97%), one practice demonstrated a much lower rate of adherence (38%).

Table 2.

Adherence to Quality Indicators

Measure	Overall		By Practice (No. yes/No. total)								Fisher's Exact Test P
			1		2		3		4
	%	No.Yes/No. Total	%	No. Yes/No. Total	%	No. Yes/No. Total	%	No. Yes/No. Total	%	No. Yes/No. Total
Path report confirming malignancy	93	243/260	96	98/102	95	58/61	94	48/51	85	39/46	.10
Dukes or AJCC staging	90	234/260	86	88/102	93	57/61	94	48/51	89	41/46	.37
CEA before surgery or chemotherapy	57	149/260	59	60/102	48	29/61	57	29/51	67	31/46	.23
CEA in 6 months after surgery or chemotherapy	83	215/260	84	86/102	82	50/61	75	38/51	89	41/46	.28
Colon evaluation within 12 months*	46	71/156	37	17/46	47	22/47	45	14/31	56	18/32	.41
Chemotherapy discussed or recommended†	96	190/197	95	83/87	92	36/39	100	35/35	100	36/36	.23
Chemotherapy consent documented‡	85	121/143	100	54/54	38	13/34	100	23/23	97	31/32	< .001
Chemotherapy flow sheet‡	98	140/143	100	54/54	97	33/34	100	23/23	94	30/32	.18

Abbreviations: AJCC, American Joint Committee on Cancer; CEA, carcinoembryonic antigen.

* Includes only patients for whom 12 months had elapsed.

† Includes only patients for whom guidelines recommend use of chemotherapy.

‡ Includes only patients who received chemotherapy.

Discussion

The initial phase of MQPI was expected to achieve three aims. First, it was expected to demonstrate the feasibility of conducting a multisite evaluation of quality of care for colorectal cancer within an existing affiliate network. Evidence this aim was achieved included the ability of the participating institutions to identify and agree on a standard set of quality indicators, identify and obtain medical records meeting eligibility criteria, conduct record abstractions in a manner that met quality standards, and forward the data to a central site for analysis via a Web-based application. It should be noted that this success reflects, in part, the selection of practices based on their willingness to participate and provide resources.

The second aim was to obtain information about adherence to eight quality-care indicators. Through a process that included multiple checks on quality control, overall and practice-specific rates of adherence were calculated for these indicators. With the notable exception of documented consent for chemotherapy, there was limited variation in adherence across practices. Similarities exist for rates of adherence reported here and those reported elsewhere. For example, both QOPI⁵ and MQPI found adherence rates of ≥ 85% across practices for the presence of a pathology report in the medical record, and use of chemotherapy flow sheets. In addition, both projects reported significant variation across practices for documentation of consent for chemotherapy. Differences, however, were also evident. While we found uniformly high (≥ 85%) adherence for staging having been completed, QOPI reported a lower range of adherence (78% to 93%) and significant variation across practices.⁵ Nevertheless, the general consistency in adherence rates is remarkable given the differences across projects in patient characteristics, case selection procedures, and practice characteristics.

The third aim was to provide a specific target or targets for quality improvement efforts. Findings suggested that one practice in particular needed to address their low rate of documented consent for chemotherapy. This problem was apparently due to this institution's lack of policies and procedures for obtaining patient consent for chemotherapy. In addition, all four practices could benefit from examining current practices for performance of baseline and follow-up CEA testing, and performance of a complete colon evaluation within 12 months of diagnosis. Ethical considerations justify improving consent rates, and empirical evidence justifies improving performance of CEA testing and complete colonoscopic evaluations.

Although CEA monitoring is not considered useful for screening purposes,⁶ evidence suggests the utility of baseline measurement at the time of diagnosis. Preoperative CEA levels have been shown to correlate with stage of disease and to be an independent prognostic indicator following curative-intent surgery.^7,8 Elevated levels of CEA are associated with reduced survival⁹ and shorter time to recurrence¹⁰ for stage II and III colorectal cancer. Preoperative levels also seem to be a significant prognostic factor for stage I disease,¹⁰ but not for stage IV metastastic disease.^8,10 Additionally, there is strong evidence to support the usefulness of CEA level in detecting disease recurrence. In patients with resected colorectal cancer, CEA levels should normalize within 4 to 6 weeks,¹¹ and persistent elevation is considered highly suggestive of residual or systemic disease.¹² It should be noted, however, that some patients may have normal preoperative CEA levels despite significant disease burden. In these cases, clinicians may chose not to conduct follow-up CEA testing since it is likely to be uninformative. The possibility that the current set of medical records included patients like this cannot be ruled out. Overall, it has been estimated that the CEA test possesses sensitivity of 77% and specificity of 98% for detection of recurrence.⁷ Given its sensitivity and specificity in detecting recurrent disease, CEA monitoring has been endorsed by both ASCO¹³ and the National Comprehensive Cancer Network.¹⁴

Between 2% and 7% of patients with colorectal cancer may harbor one or more additional synchronous cancers at the time of diagnosis.^15–17 Approximately 25% of patients may also have synchronous benign polyps.¹⁷ Based on such evidence, the American Cancer Society and US Multi-Society Task Force on Colorectal Cancer strongly recommend that the colon and rectum be carefully cleared of synchronous neoplasia¹⁸; in obstructed colons, where possible, double contrast enema or computed tomography colonography should be done preoperatively, and colonoscopy should be performed 3 to 6 months after surgery.¹⁸

With quality improvement as a goal, three types of feedback were provided to professional staff from each practice shortly after the data were tabulated. First, staff received information about the overall rate of adherence to each quality indicator. Second, they were informed of their rate of adherence to each indicator. Third, they were informed of how well the other three practices performed on each indicator; this information was presented in a blinded fashion such that each practice was unable to identify which rates belonged to the other three practices. Feedback has resulted in a quality improvement effort to improve rates of consenting for chemotherapy at the one practice with low adherence and has initiated discussion at all four practices on current patterns of care related to performance of CEA tests and complete colonoscopic examinations.

Two limitations of this initial evaluation of MQPI should be noted. First, the project was limited to patients with colorectal cancer. It remains unknown whether or not practitioners evaluated in this project demonstrated similar rates of adherence to quality care in treating other forms of cancer. However, the clinical significance of this project is underscored by the fact that colorectal cancer represents the second most common cause of cancer deaths in the men living in the United States and the third most common cause in women living in the United States.¹ Moreover, a focus on colorectal cancer permitted examination of disease-specific indicators of quality care, such as performance of CEA tests. A second limitation involved the voluntary nature of participation in the project. As a consequence, no conclusions can be drawn about the rate of adherence to the quality indicators among all members of the Moffitt Affiliate Network or, more generally, among providers of cancer treatment in Florida.

Our initial experience suggests several future directions for MQPI. One direction is to expand participation in the assessment of the quality of colorectal cancer care to all institutions in the Moffitt Affiliate Network. In addition to allowing each institution to compare itself with all other network members, such a project would provide useful information about the quality of care provided by the network as a whole. Another direction is to resurvey the original four participating institutions to see if aspects of care identified as needing improvement (eg, performance of CEA tests and consent for chemotherapy) have improved over time. Yet another direction would be to abstract additional information about the quality of colorectal cancer care from records that have already been reviewed at the four participating institutions. For example, the NICCQ⁴ includes several diagnostic and treatment indicators of the quality of colorectal cancer care, that were not part of the initial survey. By expanding the review to include NICCQ indicators, direct comparisons can be made between adherence rates for Moffitt Affiliate Network members and population-based estimates reported by the NICCQ. Finally, the potential exists to expand the current effort within the Moffitt Affiliate Network to survey the quality of care for other common cancers such as breast or prostate cancer.

Given the potential interest of many oncology practices in conducting similar evaluations, several recommendations can be offered for implementing procedures to monitor quality of care. First, the ability to conduct such evaluations is greatly facilitated by participation in organized multicenter efforts. For example, participation in the current project provided each institution with a standard set of indicators that allowed them to directly compare their performance with that of the other institutions. In addition, each institution received a training manual and instruction for office staff, designed to ensure the standardization of the abstracting procedures and, thus, the quality of the data collected. Second, practices must recognize that the measurement of quality will require an investment of resources. In the current project, each practice provided the services of office staff who reviewed medical records and entered abstracted data into a computerized database. It is estimated that these tasks took approximately 45 minutes per case to complete. Assuming $20 per hour for staff expenses, cost can be estimated at $1,500 per 100 cases reviewed. Participation in the larger effort meant that the individual practices did not have to bear other costs of conducting the quality evaluation (eg, creation of the Web portal and analysis of the data collected). It must be acknowledged, however, that in many settings, office staff may not be available or qualified to conduct record reviews. To address this issue in our affiliate network, we are currently exploring the costs and logistics of providing such practices with the services of an abstractor who would travel to the site. Third, investments in information technology have the potential to greatly facilitate the measurement of quality indicators. Conversion to electronic medical records raises the possibility that information relevant to quality assessment can be abstracted in an automated fashion at minimal cost and at more frequent intervals than is feasible with reviews of paper records.

The initial phase of MQPI demonstrated that feasibility and utility of assessing quality indicators for specific cancers, among members of an existing affiliate network of cancer treatment providers. Findings identified areas in which adherence to quality care was consistently high, but also identified areas in which adherence was less than optimal. To the best of our knowledge to date, this is the first project to accomplish these specific aims. The experience obtained has laid a valuable foundation for future efforts to expand this initiative in several directions, all of which have in common the potential to improve the quality of cancer care in a US state among the largest in population and highest in rates of death due to cancer.

Authors' Disclosures of Potential Conflicts of Interest

The authors indicated no potential conflicts of interest.