Evaluating the quality of psychosocial care in outpatient medical oncology settings using performance indicators

Abstract

Objective

An American Psychosocial Oncology Society workgroup has developed indicators of the quality of psychosocial care that can be measured through review of medical records. The present report describes the first large-scale use of these indicators to evaluate psychosocial care in outpatient medical oncology settings.

Methods

Medical records of 1660 colorectal, breast and non-small cell cancer patients first seen by a medical oncologist in 2006 at 11 practice sites in Florida were reviewed for performance on indicators of the quality of psychosocial care.

Results

Assessment of emotional well-being was significantly less likely to be documented than assessment of pain (52 vs 87%, p<0.001). A problem with emotional well-being was documented in 13% of records and evidence of action taken was documented in 58% of these records. Ten of eleven practice sites performed below an 85% threshold on each indicator of psychosocial care. Variability in assessment of emotional-well being was associated (p<0.02) with practice site and patient gender and age while variability in assessment of pain was associated (p<0.001) with practice site and cancer type.

Conclusions

Findings illustrate how use of the psychosocial care indicators permits identification of specific practice sites and processes of care that should be targeted for quality improvement efforts. Additionally, findings demonstrate the extent to which routine assessment of emotional well-being lags behind routine assessment of pain in cancer patients.

Introduction

A 2007 Institute of Medicine (IOM) report, titled ‘Cancer Care for the Whole Patient: Meeting Psychosocial Health Needs,’ summarized the current status of efforts to provide psychosocial care for people with cancer [1]. The report concluded that, despite evidence of the effectiveness of psychosocial services, many patients do not receive help for problems that would benefit from this type of care [1]. The report also described how failure to address these problems results in needless patient and family suffering, contributes to reduced adherence to prescribed treatment regimens, and can potentially affect the course of disease [1]. The reasons for this failure are many and include the tendency of oncology care providers to underestimate distress in patients [2] and not link patients to appropriate services when needs are identified [3]. To address these problems, the report recommended that provision of appropriate psychosocial services be adopted as a standard of quality cancer care [1]. In addition, the report identified a model for effective delivery of psychosocial services that includes implementation of processes for identifying patients’ psychosocial needs and then designing and implementing plans that links patients with needed psychosocial services [1].

These recommendations are similar to those in the Clinical Practice Guidelines for Distress Management first issued by the National Comprehensive Cancer Network (NCCN) in 1999 [4] and updated annually [5]. Similar to the IOM report [1], the NCCN guidelines recommend that all patients undergo screening routinely to identify the level and source of their distress [4]. The specific services and resources subsequently recommended are designed to be appropriate to the nature and severity of the problems identified through screening and, if indicated, further evaluation. Relatively little is known about the extent to which NCCN distress management guidelines are being followed. One source of evidence is a 2005 survey of 15 NCCN member institutions [6]. Although psychosocial services were available at all 15 institutions, only 8 (53%) were conducting routine screening for distress [6]. Among these 8 institutions, 3 (37.5%) reported screening all patients routinely and 5 (62.5%) reported screening only certain patients routinely (e.g. transplant candidates) [6]. Of the 8 institutions conducting any routine screening, 7 (87.5%) reported that patients identified as distressed were routinely referred to a mental health professional [6]. Another source of information is a 2004-5 survey of 448 oncologists who are members of the American Society of Clinical Oncology (ASCO) [7]. Sixty-five percent reported they screened their patients for distress routinely.

Taken together, the IOM report and these surveys suggest the need to foster greater implementation of recommendations for the psychosocial care of cancer patients. One way to accomplish this goal would be to give practitioners feedback about the quality of the psychosocial care they provide their patients. To evaluate this possibility first requires development of measurable indicators of the quality of psychosocial care. In the absence of accepted indicators, the American Psychosocial Oncology Society (APOS) formed a quality of care workgroup in 2007. The workgroup’s initial focus has been on developing process measures of the quality of psychosocial care that can be evaluated via medical record abstraction. Following a review of the relevant literature, indicators were developed to measure two components considered necessary (but not sufficient) for providing quality psychosocial care: a process for identifying distressed patients and a process for linking distressed patients with services.

The current report describes the first large-scale use of these indicators to evaluate the quality of psychosocial care in outpatient medical oncology settings. Using the resources of the Florida Initiative for Quality Cancer Care (FIQCC), the psychosocial indicators were embedded in a larger set of quality indicators and data were collected at 11 practice sites in the state of Florida. As described elsewhere [8], the FIQCC is a regional effort designed to identify and address the barriers to delivering quality cancer care. We have previously published an initial evaluation of the psychosocial care indicators based on a subset of the colorectal cancer patients [9]. The primary aims of this report were to describe performance on these indicators in a large sample of outpatients with colorectal, breast, and non-small cell lung cancer and to identify sources of variation in performance across patients. Factors examined as potential sources of variation were practice site and patient age, gender, and cancer type. A secondary aim was to determine whether performance rates and sources of variation in performance were similar for assessment of emotional well-being and assessment of pain.

Methods

Study sites

The FIQCC consists of the following medical oncology practices at or affiliated with: Space Coast Medical Associates (Titusville), Center for Cancer Care and Research (Lakeland), Florida Cancer Specialists (Sarasota), Ocala Oncology Center (Ocala), Robert and Carol Weissman Cancer Center (Stuart), Cancer Centers of Florida (Orlando), Tallahassee Memorial Cancer Center (Tallahassee), University of Florida Shands Cancer Center (Gainesville), Mayo Clinic Cancer Center (Jacksonville), and Moffitt Cancer Center (Tampa). Each practice met the following criteria for participation in the initiative: (1) medical oncology services provided by more than one oncologist; (2) availability of a medical record abstractor; and (3) estimate of 40 or more cases each of colorectal, breast, and non-small cell lung cancer for calendar year 2006 based on available statistics. The project received approval from Institutional Review Boards at each institution. To maintain patient privacy, records were coded with a unique project identifier prior to transmission to the central data collection site. Based on exempt status, informed consent from patients was not required to access medical records.

Quality indicators

Medical records were abstracted for numerous indicators of the quality of cancer care [8]. The present report focuses on two indicators of the quality of psychosocial care: (1) there should be evidence in the medical record that the patient’s current emotional well-being was assessed within 1 month of the patient’s first visit with a medical oncologist; and (2) if a problem with emotional well-being was identified, there should be evidence in the patient’s medical record that action was taken to address the problem or an explanation provided for why no action was taken. Measurement was operationalized by formulating three questions that could be answered yes or no based on medical record review (Table 1). In addition to these indicators, the current report includes information about whether there was evidence that the patient’s pain status was assessed within 1 month of the first visit with a medical oncologist (Table 1).

Table 1. Quality indicators and related rating criteria.

Is there evidence current emotional well-being was assessed within one month of the first visit with a medical oncologist? Documentation is sufficient if the medical chart includes: copy of distress, depression, or anxiety screening measure OR copy of form including patient self-report of distress, depression, or anxiety OR note with statement referring to current ‘coping,’ ‘adjustment,’ ‘distress,’ ‘emotional,’ ‘depression,’ or ‘anxiety’ status of patient If Yes, mark Yes If No, mark No Was a problem with emotional well-being identified within one month of the first visit with a medical oncologist? A problem can be considered present if the medical chart includes a statement that the patient is ‘distressed,’ ‘depressed,’ ‘anxious,’ or having problems with ‘coping,’ ‘adjustment,’ or ‘emotional well-being.’ If Yes, mark Yes If No, mark No Is there evidence action was taken to address the problem or an explanation provided of why no action was taken within one month of the first visit with a medical oncologist? Documentation is sufficient if the medical chart includes one of the following: note describing care provided by primary oncology team for problem with ‘coping,’ ‘adjustment,’ ‘depression,’, ‘anxiety’, or ‘distress’ OR note describing referral to another professional for care of problem with ‘coping,’ ‘adjustment,’ ‘depression,’ ‘anxiety,’ or ‘distress’ OR note describing referral to mental health professional (i.e. psychiatrist, psychologist, social worker, pastoral care professional, mental health counselor, or psychotherapist) OR note describing care provided by another professional for problem with ‘coping,’ ‘adjustment,’ ‘depression,’ ‘anxiety,’ or ‘distress’ OR note describing why no action was taken to address problem with ‘coping,’ ‘adjustment,’ ‘depression,’ ‘anxiety,’ or ‘distress’ If Yes, mark Yes If No, mark No Is there evidence that pain was assessed within one month of the first visit with a medical oncologist? Documentation is sufficient if the medical chart includes a note with a statement referring to any ‘pain’ or ‘discomfort’ If Yes, mark Yes If No, mark No

Medical record selection

All patients age 18 or older at each practice site diagnosed with colorectal, breast, or non-small cell lung cancer and seen for a new medical oncology consultation in calendar year 2006 were eligible for case selection with certain disease-specific exclusions (e.g. for non-small cell lung cancer, patients with mixed small and non-small cell disease and carcinoid tumors were excluded). Preliminary estimates suggested the number of colorectal cancer cases would be smaller than breast and non-small cell lung cancer cases. Consequently, the following strategy was used to obtain similar numbers of cases of each disease at each practice site. After first determining the number of colorectal cancer cases abstracted, cases of breast and non-small cell lung cancer were randomly selected from available cases. For sites that abstracted 60 or fewer cases of colorectal cancer, up to 60 cases each of breast and non-small cell lung cancer were abstracted (if possible). For sites that abstracted more than 60 cases of colorectal cancer, the number of breast and non-small lung cancer cases abstracted was set at the number of colorectal cancer cases abstracted.

Medical record review

A manual for data identification, abstraction, and entry was developed to ensure consistency across study sites. An experienced medical record abstractor was designated the chief abstractor and had responsibility for training and monitoring data abstractors at each practice site. Training and monitoring was conducted in three phases. The initial phase consisted of on-site training. Medical records of approximately 15–20 cases from a previous year were retrieved at each site. After the chief abstractor demonstrated the rating of each quality indicator, the site abstractor and chief abstractor then independently rated four to five charts, compared ratings, and discussed any discrepancies. This process continued until the two abstractors agreed on an average of 70% or greater of their ratings for the larger set of indicators for each disease over five consecutively rated charts. The second phase occurred after each site completed abstraction of approximately one-third of their 2006 cases for each disease. The chief abstractor independently rated approximately five cases of each disease that had been rated and submitted by each site abstractor. Discrepancies were discussed and resolved and relevant portions of the training and reference manual were reviewed if necessary. In the third phase, the same procedures were used when approximately two-thirds of the 2006 cases for each disease had been abstracted at a site.

Statistical analysis

Descriptive statistics were used to summarize patient demographic and clinical characteristics. Overall and site-specific performance rates (percentages) were calculated for each indicator. Overall performance on the emotional well-being and pain assessment indicators were compared by computing the exact p value for McNemar’s test. The contribution of age, gender, cancer type, and practice site individually to performance on the emotional well-being assessment indicator was evaluated by conducting bivariate logistic regression analyses. A multivariable logistic regression was then conducted to identify the relative contributions of each of these variables. Similar analyses were conducted for performance on the pain assessment indicator. The odds ratio and 95% confidence interval (CI) for each variable were estimated from the logistic regression models. All analyses were performed with a p<0.05 significance level (two-tailed).

Results

Case characteristics

Medical records of 1660 patients were abstracted. The number of cases per practice site ranged from 51 to 291 (median = 153). Sixty-seven percent of patients (n = 1116) were female and 33% (n = 544) were male. Mean age was 64 years (SD = 13; range = 22-97). Thirty-one percent (n = 507) were diagnosed with colorectal cancer, 37% with breast cancer (n = 622), and 32% with non-small cell lung cancer (n = 531). Of 1551 patients for whom disease stage was available, 71% had non-metastatic disease (n = 1107) and 29% had metastatic disease (n = 444). Information about site-specific performance on the emotional well-being and pain quality indicators appears in Figures 1–3. Rates for each site (lettered A–K) are presented in masked form to preserve site anonymity per prior agreement with the participating institutions.

Figure 1. Emotional well-being assessed.

Figure 3. Action taken for emotional problem.

Performance on the assessment indicators

The median practice site performance rate for assessment of emotional well-being and assessment of pain were 47 and 88%, respectively. Across sites, emotional well-being was assessed in 52% of cases (n = 861) and pain was assessed in 87% of cases (n = 1438). This difference in indicator percentages is statistically significant (p<0.001).

Relationship of demographic and clinical variables with assessment of emotional well-being

Bivariate logistic regression analyses were conducted to evaluate the relationship of age, gender, cancer type, and practice site with assessment of emotional well-being. Findings indicated that age was significantly negatively related to assessment of emotional well-being (p<0.001; OR = 0.99, 95% CI = 0.98–0.99). The magnitude of this relationship reflects a 1.5% decrease in assessment of emotional well-being for each year increase in age. Findings also indicated a significant relationship of gender with assessment of emotional well-being (p = 0.002; OR = 0.72, 95% CI = 0.59–0.88). This relationship is reflected in rates of assessment of emotional well-being of 46% for males and 55% for females. The relationship of cancer type with assessment of emotional well-being was not statistically significant (p = 0.53). Consistent with this finding, rates of assessment of emotional well-being were 51% for colorectal cancer, 53% for breast cancer, and 50% for non-small cell lung cancer. Finally, there was a significant relationship of practice site with assessment of emotional well-being (p<0.001). This finding is consistent with results showing that performance rates on this indicator ranged from 12 to 86% across practice sites (Figure 1).

Results of a multivariable logistic regression analyses indicated that age, gender, and practice site were independently significantly associated with assessment of emotional well-being (Table 2).

Table 2. Results of bivariate and multivariable analyses of predictors of assessment of emotional well-being and pain.

	Emotional well-being		Pain
Variable	Bivariate p value	Multivariable p value	Bivariate p value	Multivariable p value
Age	<0.001	0.003	0.64	0.56
Gender	0.002	0.01	0.02	0.49
Cancer type	0.53	0.99	<0.001	<0.001
Practice site	<0.001	<0.001	<0.001	<0.001

Relationship of demographic and clinical variables with assessment of pain

Bivariate logistic regression analyses were also conducted to evaluate the relationship of age, gender, cancer type, and practice site with assessment of pain. Findings indicated that age was not related to assessment of pain (p = 0.64; OR = 1.0, 95% CI = 0.99-1.01). In contrast, there was a significant relationship of gender with assessment of pain (p = 0.02; OR = 1.44, 95% CI = 1.05–1.99). This relationship is reflected in rates of assessment of pain of 89% for males and 85% for females. The relationship of cancer type with assessment of pain was also statistically significant (p<0.001). Additional analyses indicated that rates of assessment for non-small cell lung cancer were significantly higher than rates for colorectal cancer (p<0.001; OR = 0.46, 95% CI = 0.31–0.69) and breast cancer (p<0.001; OR = 0.42, 95% CI = 0.29–0.62). Consistent with these findings, rates of assessment of pain were 85% for colorectal cancer, 83% for breast cancer, and 92% for non-small cell lung cancer. Finally, there was a significant relationship of practice site with assessment of pain (p<0.001). This finding is consistent with results showing that performance rates on this indicator ranged from 55 to 97% across practice sites (Figure 2).

Figure 2. Pain assessed.

Results of a multivariable logistic regression analyses indicated that cancer type and practice site were independently associated with assessment of pain (Table 2). Gender, which yielded a significant finding in the bivariate analyses, did not demonstrate a significant association in the multivariable model.

Performance on the action taken indicator

A problem in emotional well-being was documented in medical records of 214 patients. This figure represents 13% of the 1660 records reviewed and 25% of the 861 records where there was documentation that emotional well-being was assessed. Among these 214 patients, the median practice performance rate for evidence that action was taken to address the problem (or an explanation provided for why no action was taken) was 57% (range = 13–100%, Figure 3). Across sites, there was evidence for action being taken (or an explanation provided for no action) in 58% of cases (n = 125). Analyses examining the relationships of age, gender, cancer type, and practice site with performance on this indicator were not conducted due to the limited number of patients identified as having a problem with emotional well-being.

Discussion

There was wide variability across practice sites on the two indicators of the quality of psychosocial care. Rates ranged from 12 to 86% for assessing emotional well-being and 13 to 100% for taking action to address an identified problem. Performance below 85% has been used previously as a threshold for identifying opportunities to improve the quality of cancer care [10]. Ten of the eleven sites performed below this threshold for assessing emotional well-being and for taking action to address a problem in emotional well-being. Among sites performing below 85%, rates also varied widely. With regard to assessing emotional well-being, findings suggest that two sites (i.e. those with rates of 73 and 74%) had some procedures in place for systematically assessing problems in emotional well-being. In the seven remaining sites where rates ranged from 12 to 57%, findings suggest the absence of procedures for systematically assessing emotional well-being. With regard to addressing problems in emotional well-being, findings suggest that the site with a 13% performance rate had few procedures in place to address problems. Among the nine remaining sites where rates ranged from 35 to 73%, results suggest that procedures were in place but were not being routinely implemented.

These results contrast sharply in performance on the pain indicator. Although five of the eleven sites performed below 85%, four of these sites achieved rates just below the threshold (i.e. between 82 and 84%). This pattern is consistent with results showing that the average performance rate for assessment of pain (87%) was significantly higher than the corresponding rate for emotional well-being (52%). Taken together, the findings demonstrate that routine symptom assessment was much more likely to occur for pain than for emotional well-being. This state of affairs suggests that, to date, efforts to promote routine assessment of pain in cancer patients [11] have been more successful than efforts to promote routine assessment of emotional well-being [4]. The higher rates of pain assessment also likely reflect the effects of requirements by some hospital accreditation bodies, such as the Joint Commission on Accreditation of Health Care Organizations [12], that pain be assessed routinely in all patients.

In addition to comparing rates for assessment of pain and emotional well-being, the present study examined whether practice site and patients’ age, gender, and cancer type were related to indicator performance. Results of multivariable analyses demonstrated similarities and differences regarding the relative contributions of these variables to assessment of pain and emotional well-being. Although the range of performance rates across sites was narrower for pain (55–97%) than emotional well-being (12–86%), practice site was associated with both forms of assessment. That is, the likelihood that a patient’s pain or emotional well-being would be assessed was, in part, attributable to the practice site where they were seen. Beyond this similarity, results diverge. For emotional well-being, there were significant associations also with age and gender, with younger patients and female patients more likely to be assessed. In contrast, cancer type was the only other variable significantly associated with pain assessment, with lung cancer patients more likely than breast or colorectal patients to have their pain assessed.

The associations with age and gender in the present study are consistent with previous research on factors related to heightened distress in cancer patients. For example, a study of emotional distress in 3071 cancer outpatients found that younger age and female gender, but not cancer type, were significant predictors of clinically significant distress [13]. In contrast, a study of 1429 cancer patients did not find age or gender to be significant predictors of pain [14]. These features raise the possibility that associations of age and gender with assessment of emotional well-being in the present study are due to clinicians documenting assessments more often in patients who appear more distressed or who express more distress. If true, findings suggest that assessment of well-being is being driven to a certain extent by patients concerns coming to the attention of clinicians rather than through systematic screening. The association of cancer type with assessment of pain is less easily understood. One possible explanation is that lung cancer patients were more likely than breast or colorectal cancer patients to report pain due to their having more severe disease. The lack of data on performance status and the incomplete data about disease stage and metastatic status precludes an examination of this possibility.

Findings from the present study identified practice sites where processes currently in place for assessing and addressing problems in emotional well-being were less than optimal. To convey this information, a report was provided to each practice that summarized performance on these and other quality indicators using the format shown in Figures 1–3. Each practice was able to determine how its performance compared with every other practice individually and to the average of all other practices. Providing feedback in this manner has the potential to yield improvements in the quality of psychosocial care. Evidence in support of this view comes from the Quality Oncology Practice Initiative (QOPI), a voluntary medical oncology practice-based project sponsored by ASCO [15]. In 2008, QOPI added the psychosocial care indicators to its core set of measures completed by all participating practices. As part of a summary report, each practice was informed of its performance on these and other indicators relative to all other participating practices. Among 166 practices that participated in two consecutive annual audits, assessment of emotional well-being was found to improve significantly over time from 64% in Fall 2008 to 73% in Fall 2009 [16]. These improvements occurred in the absence of any intervention other than providing performance feedback.

While provision of feedback alone may yield some improvements in the quality of psychosocial care, a more active approach could yield greater results. One method would be for practices to examine their processes for assessing emotional well-being and compare them with their processes for assessing pain, given the generally higher rates of performance for the latter. In addition, practices might share information among themselves about processes used successfully to assess and address problems in emotional well-being. The extent, however, to which ‘best processes’ among oncology practices participating in quality audits can be considered ‘best’ from an evidence-based perspective is unknown. For example, methods used to assess distress may involve informal questioning conducted during a clinical examination, an approach likely to lead to under-recognition of psychological distress when compared with use of standardized screening measures [2]. Two future directions can be identified that might facilitate greater progress in implementing evidence-based psychosocial care.

First, the existing psychosocial care indicators can be revised to permit more exact comparisons of the processes in place with those recommended by clinical practice guidelines. To be more consistent with NCCN distress management guidelines, for example, the indicator regarding assessment of emotional well-being could be revised to specify that screening be performed using a validated screening measure. Similarly, the indicator regarding action being taken to address a problem in emotional well-being could be revised to specify that patients receive certain forms of care shown to be effective for their problems. Second, practices seeking to improve psychosocial care can be provided with care models consistent with clinical practice guidelines to guide them. Practices will also need guidance about how to implement new care models and how to sustain the changes made. These objectives can be achieved by conducting a series of rigorously designed demonstration projects that seek to improve psychosocial care in ‘real world’ oncology settings. To be considered successful models, these projects should yield disseminable methods that result in sustained improvements in the practice’s delivery of psychosocial care and in the emotional well-being of its patients.

Several limitations of the current study should be noted. First, the degree of inter-rater agreement was not assessed separately for the psychosocial care indicators; therefore, the reliability of these ratings is unknown. Second, no ‘gold standard’ measure of emotional well-being was routinely administered to all patients whose medical records were reviewed. Therefore, the accuracy of the methods used at the practice sites to identify patients with problems with emotional well-being is unknown. Third, medical records were not examined to evaluate the outcomes of actions taken to address problems with emotional well-being. Consequently, the adequacy of these actions in resolving problems is unknown. Fourth, the quality indicators focus only on the 1-month period after a patient’s initial visit with a medical oncologist. Consequently, they do not address recommendations that patients’ emotional well-being be assessed at appropriate intervals throughout the course of treatment [5]. Despite these limitations, evidence that the indicators demonstrate variability across practice sites and identify processes that can be targeted for improvement support their continued use. Clinicians and researchers are encouraged to report additional information about their utility in measuring the quality of psychosocial care and guiding quality improvement efforts.