Cancer Patient Assessment and Reports of Excellence: Reliability and Validity of Advanced Cancer Patient Perceptions of the Quality of Care

Abstract

Purpose

Consumer perceptions are important measures of the quality of cancer care. This article describes the validation of new measures of the quality of cancer care at the time of diagnosis and treatment for advanced cancer with life-limiting prognosis.

Methods

Focus groups, review of guidelines, and an expert panel were used to construct two surveys of the quality of cancer care. A prospective cohort study examined the reliability and validity of three problem scores (ie, counts of the opportunities to improve the quality of care) that examine care at the time of diagnosis and initial treatment.

Results

At the first interview, 58% of 206 cancer patients (54.9% females; 27.5% with lung cancer; 5.4% with pancreatic cancer; 30.4% with colorectal cancer; 18.6% with breast cancer; mean age, 66.6 years) identified one or more concerns with communication about being diagnosed with advanced cancer. At the second interview, 57.0% of the respondents voiced one or more concerns about treatment communication, and 30.2% expressed one or more concerns about the experience of treatment. Each of the problem scores demonstrated both internal consistency with Cronbach's α > .75 and short-term stability of responses in a subsample that had the survey administered twice in 72 hours. Factor analysis largely confirmed the proposed scale structure. All three measures demonstrated moderate correlations suggesting evidence of construct validity.

Conclusion

The three proposed problem scores demonstrate evidence of reliability and validity that warrants further testing to examine their responsiveness and discriminate validity in larger, more generalizable samples.

INTRODUCTION

Donabedian¹ noted that, “achieving and producing health and satisfaction, as defined for its individual members by a particular society or subculture, is the ultimate validator of the quality of care.” Increasingly, patients' and family members' perceptions have been recognized as important outcomes of the quality of cancer care. The measurement of consumer perceptions with typical satisfaction measures (eg, completely satisfied to completely dissatisfied) has raised important methodologic concerns. Williams and Calnan² found that 95% of persons were satisfied with medical care, yet 38% reported that they had difficulty discussing personal problems with their physician and 35% felt that physicians did not spend enough time with them. In response to concerns with satisfaction measures, Cleary et al³ created patient-centered reports, where questions were framed to ask the patient or family member to report as an expert witness on a process of care and whether that process met a certain standard (eg, “Were you told the purpose of your medications in a way that you could understand?”).

Previously, Teno and colleagues developed a survey, the Family Evaluation of Hospice Care, using patient-centered reports to examine bereaved family member perceptions of the quality of dying. This current research builds on that work to develop a survey that reflects the concerns of patients diagnosed with advanced cancer. In this research effort, we used a structured literature review of applicable guidelines, focus groups with consumers, and the advice of an expert panel to develop two surveys that focused on critical time points in the disease trajectory—the experience of being diagnosed with an advanced stage of cancer and beginning treatment for the cancer. In this study, we report the reliability and validity of three measures developed to measure the quality of medical care received at the time of diagnosis of advanced-stage cancer and during initial treatment.

METHODS

Informed by focus groups, an expert panel, and review of existing guidelines,^4–16 we defined high-quality cancer care based on institutions meeting the following needs of cancer patients: health care providers sensitively communicate with patients about their prognosis and promote shared decision making to the extent that patients want to be involved; patients are educated about what they can expect during treatment, what to monitor, and who they can turn to for help; patients are provided with their desired emotional support and symptom amelioration; patients are treated with respect; cancer treatment sessions support comfort, ensure privacy, and promote healing; and care is coordinated among health care providers. Two of these needs (patients are provided with their desired emotional support and symptom amelioration and patients are treated with respect) have existing measures.¹⁷

An expert panel reviewed each item in the developed surveys for relevance to high-quality cancer care and whether the process of care was under the control of a health care provider (either the physicians or other members of their staff). The survey underwent cognitive testing with modification or deletion of items that were not clear to the respondents.

Data Collection

After a pretest in Rhode Island, both surveys were tested on a prospective cohort of persons diagnosed with cancer at an advanced stage that implied a life-limiting illness. Cancer patients with the following diagnoses were recruited from oncologist offices in New Hampshire, Connecticut, and Rhode Island: stage IV pancreatic cancer; inoperable or stage IV non–small-cell lung cancer; stage IV breast cancer that failed first-line treatment; stage IV skin cancer; and stage IV genitourinary cancers that failed first-line treatment. Office staff identified potential patients who were screened by the oncologist, and then patient permission was requested to release their information to researchers at Brown University or Dartmouth Medical School. If that permission was granted, a research assistant either met or called the patient to inquire if the patient would participate in the study. Only 99 of the 299 eligible patients at the Dartmouth site chose to participate because there was another study simultaneously recruiting patients for a randomized controlled trial of a palliative care intervention. At the Brown University data collection site, 119 of the 180 eligible cancer patients agreed to participate in the survey.

The first survey was administered shortly after the patient was diagnosed with or progressed to an advanced stage of cancer. A second survey was administered 1 to 2 months later when the patient had completed at least one treatment cycle. Twenty-three respondents volunteered to have the survey administered again 48 to 72 hours later so we could examine the short-term reliability.

Two types of response tasks were used to measure patients' perceptions of the quality of cancer care at each time period. The first type relied on the methodology of patient-centered reports, which ask patients to report on their perceptions of the quality of care using response categories of “yes/no” or a four-point scale of “always,” “usually,” “sometimes,” and “never.” The second type of response task measured patient unmet needs by rating a need as “less than was needed,” “more than was needed,” or “just the right amount.”

Because high-quality medical care strives for excellence, responses to each question were recoded to create dichotomous measures that indicated opportunities to improve (ie, concerns with care). For example, responses to the question “How often are your cancer care providers available to speak with you when you have a concern about your chemotherapy?” ranged from “always” to “never.” For this particular question, we considered an answer of anything other than “always” as an opportunity to improve and thus dichotomized the variable by assigning a value to 0 to “always” and a value of 1 to all other responses. For the unmet need questions, responses of “less than was needed” and “more than was needed” were considered opportunities to improve and assigned a value of 1. See columns 1 and 2 of Table 1 for item descriptions and rates of concerns. This scoring decision reflects the intended use of the tool for quality improvement purposes.

Table 1.

Survey Items and Rates of Opportunities to Improve That Compose the Three Proposed Problem Scores

Problem Score and Survey Items	Frequency of Opportunity to Improve (%)
Time 1: Communication at the time of diagnosis problem score
How much information did you receive about your prognosis?	13.8
Was your prognosis explained to you in a way that you could understand?	5.4
Was your prognosis presented to you in a sensitive manner?	6.4
How much emotional support did you receive from your oncologist or cancer care providers when your prognosis was discussed with you?	10.3
Were you allowed to ask as many questions as you wanted about the prognosis of your cancer?	5.9
How often have you been given confusing or contradictory information about your prognosis from your oncologist or other cancer care providers?	14.4
How often does your oncologist or cancer care provider explain your test results in a way that you can understand?	14.6
How often do you want more information about your test results than you receive from your cancer care providers?	35.4
When you see other physicians involved in your cancer care (eg, primary care physician, surgeon, radiation oncologist), how often are they aware of your test results?	28.3
How often have you been given confusing or contradictory information about your test results from your cancer care providers and/or other physicians involved in your care (eg, primary care, surgeon, radiation oncologist)?	14.6
How comfortable do you feel asking questions about your cancer treatment plan?	10.5
Do you feel that your oncologist or cancer care providers listen to your concerns about your treatment plan?	2.0
How often do your cancer care providers allow your family/friends to ask as many questions as they want about your treatment plan?	3.0
How much emotional support do your cancer care providers give to your family/friends?	7.6
How often would you say that your oncologist and cancer care providers work together as a team?	16.3
How often do you feel that your oncologist or cancer care providers are not paying enough attention to your care?	13.2
How often do you feel that you and your family are left on your own to make sure that the right things get done for your cancer care?	14.7
Have you received clear instructions from your cancer care providers on whom you should call after regular office hours and/or over the weekend?	12.8
How often were you able to reach your oncologist or his/her coverage in a timely manner?	15.6
How often have you had to wait too long to see one of your cancer care providers when you have had an urgent medical problem?	7.8
Time 2: Communication at the time of treatment decision making problem score
How much information do your cancer care providers give you about what you can do to manage side effects from your chemotherapy treatments?	6.9
How confident are you that you know what to do to manage and/or prevent the side effects from your chemotherapy treatments?	37.1
How much information do your cancer care providers give you about what you should monitor after your chemotherapy treatments?	8.3
How much information have you received from your cancer care providers about chemotherapy and/or radiation treatments?	3.8
How much information have you received from your cancer providers about the possible risks of chemotherapy and/or radiation treatments?
How much information have you received from your cancer providers about the benefits of chemotherapy and/or radiation treatments?	3.8
How often do you feel that your cancer care providers listen to your concerns about the possible side effects of chemotherapy and/or radiation treatments?	16.6
How much information have you received from your cancer care providers about possible changes in your physical appearance as a result of chemotherapy treatments?	6.3
Have you received clear instructions from your cancer care providers on whom you should call after regular office hours and/or over the weekend?	11.3
How often were you able to reach your oncologist or his/her coverage in a timely manner?	13.7
How often have you had to wait too long to see one of your cancer care providers when you had an urgent medical problem?	11.9
How often do your oncologist or cancer care providers work together as a team?	19.0
How often would you say that your oncologist and other physicians involved in your cancer care (eg, primary care physician, surgeon, radiation oncologist) work together as a team?	22.5
How often do you feel that your cancer care providers are not paying enough attention to your care?	11.4
How often do you feel that you and your family are left on your own to make sure that the right things get done for your cancer care?	14.6
How often do your cancer care providers explain what they are going to do in a way that you can understand?	17.4
Time 2: Cancer patient experience of treatments problem score
How often do your cancer care providers do everything they can to promote your comfort during treatment?	10.7
How often are your cancer care providers available to speak with you when you have a concern about your chemotherapy?	10.1
How often do the chemotherapy treatment rooms have adequate space so that your family can be with you?	14.7
Overall, do the chemotherapy treatment rooms provide you with enough privacy?	6.4
How often is everything done to make you as comfortable as possible while you are receiving your chemotherapy treatment?	9.0
How much support do you get from the staff during chemotherapy treatments?	1.9
How often do the staff promote your dignity during chemotherapy treatments?	5.8
Does the physical environment of the treatment room allow you to focus on healing and recovering from your cancer?	3.3
Overall, how often have you been treated with respect by your cancer care providers?	5.7

Measures: Problem Scores

Three problem scores were created by summing the number of concerns across various items. The first problem score focused on the time of diagnosis of advanced cancer. It consisted of 20 items that focused on communication surrounding the diagnosis of advanced cancer, prognosis, and the initial staging (or restaging) work-up. We created the other two problem scores using the second survey. They consisted of a 15-item score focusing on communication about treatment decision making and a 10-item scale regarding respondents' experiences with chemotherapy or radiation treatment. Higher scores indicated more opportunities to improve the quality of care (ie, more concerns with care). See column 1 of Table 1 to examine the particular items that went into each problem score.

Several additional questions were included for descriptive purposes and for examining the validity of the proposed problem scores. Respondents answered questions regarding the quality of their oncologists' care and their own quality of life using a 5-point likert scale ranging from “excellent” to “poor.” We included the five-item Mental Health Inventory (MHI-5), which has known reliability and validity.¹⁸ We also asked respondents to rate particular aspects of cancer care on a scale of 0 (indicating worst care) to 10 (indicating best care). Finally, we collected limited information about particular symptoms, as well as the frequency, severity, and distress of those symptoms.

Analysis

A correlation analysis examined the psychometric properties of the three proposed problem scores. Cronbach's α was used to examine the internal consistency for each proposed problem score. Depending on the distribution of the problem score, we examined the interitem and item-to-total correlations with either a Pearson or a Spearman correlation coefficient. Factor analysis was used to determine whether items were loaded on the a priori–specified domains of our conceptual model.

We assessed the criterion validity of the problem scores by correlating the problem scores with each of two measures of the rating of the quality of care. The first measure was a single question that examined overall satisfaction using a 5-point scale that ranged from “excellent” to “poor.” We expected to find a moderate correlation with this question based on its use in other validation studies¹⁹ and on the knowledge that problem scores asking about a particular aspect of care may not be the key determinant to overall satisfaction. The second measure was a multi-item scale that rated various aspects of the quality of cancer care using a 0 to 10 rating scale from “worst” to “best possible care” and summed across all items. This scale was based on the work of Teno and colleagues in creating the Family Evaluation of Hospice Care.

As a measure of construct validity, we used the Kruskal-Wallis rank sum test to examine whether respondents with more concerns with each problem score had higher levels of emotional distress as measured by the MHI-5²⁰ on the second interview. The same test was used to determine whether the problem scores differentiated the quality of care among patients who received their care either in a comprehensive cancer setting or a private office. Finally, the bivariate associations of each problem score were examined with age, education, race, and sex.

RESULTS

A total of 206 cancer patients participated in the first survey that focused on the time of diagnosis. Participants did not differ from nonparticipants in age or race, but there was a slight tendency for more women to participate in the survey (52.9% women v 48.5% men, P = .34). Table 2 lists the characteristics of sample participants. Of the 206 time 1 respondents, 162 patients participated in a second interview 2 months later, 22 patients either died or were too near death to be interviewed, 14 patients refused to be interviewed, two patients left the practice, and the remaining six patients did not participate for other reasons. Twenty-three respondents agreed to retake the survey between 48 and 72 hours with the finding of an overall agreement of 92.1% on the survey items that compose the three proposed problems scores.

Table 2.

Patient Characteristics

Characteristic	% of Patients Interviewed (N = 206)
Age, years
Mean	66.6
Standard deviation	11.6
Male	45.1
White	95.1
Education level
Eighth grade or less	1.0
Some high school but did not graduate	8.9
High school graduate or GED	36.6
1-3 years of college	29.2
4-year college graduate	11.9
More than a 4-year college degree	12.4
Type of cancer
NSCLC, stage IIIB	4.9
NSCLC, stage IV	22.6
Pancreatic, stage IV	5.4
Colorectal, stage IV	30.4
GU-prostate, stage IV	5.9
Breast, stage IV	18.6
GU-urinary bladder, stage IV	3.4
GU-renal pelvis and ureter penile, stage IV	1.5
Other	4.4
Health status
Excellent	9.4
Very good	25.6
Good	38.4
Fair	17.2
Poor	9.4
Chose active treatment	95.6
Symptoms
Pain	55.1
Trouble breathing	35.3
Anxiety	44.6

Abbreviations: GED, General Education Development Test; NSCLC, non–small-cell lung cancer; GU, genitourinary.

As was indicated earlier, Table 1 lists the frequency of concerns for each item that composed the three problem scores. Table 3 lists a number of psychometric properties of the three problem scores. The first set of columns focuses on degree of missing data and the range of reported concerns for each item that composed the problem score. Column 4 shows the average item score (ie, the rate of concern on each question divided by the total number of questions in the problem score), indicating for each item that, on average, approximately one in 10 respondents voiced a concern. The fifth column of Table 3 shows that the majority of respondents identified at least one concern in each of the first two problem scores, whereas less than one third voiced a concern about any item in the third problem score.

Table 3.

Psychometric Properties

Problem Score	Description of the Frequency of Items That Composed Each Problem Score and Examination of the Average Reported Rate of Concerns Among Cancer Patient Receiving Their Care at Comprehensive Cancer Center v Community-Based Oncologist Office							Cronbach's α	Factor/Correlation Analysis				Criterion Validity: Single Item Measure	Criterion Validity: 0-10 Multi-Item Rating Scale
	Score Range	% Missing		% Reporting Concern Across Each Item		Comprehensive Cancer Center (average % concern) (n = 89)	Private Practice (average % concern) (n = 117)		Item With Total		Item With Item
		Average	Range	Average	Range				Mean	Range	Mean	Range
Communication at the time of diagnosis problem score	0-20	1.6	0-5.8	12.4	1.9-35.4	17.3	9.1*	.84	0.41	0.160-0.563	0.22	0.004-0.863	0.41†	–0.53†
Communication at the time of treatment decision making problem score	0-15	2.5	1.3-6.9	13.0	3.8-36.9	18.5	9.7‡	.78	0.37	0.178-0.549	0.16	−0.001-0.460	0.36†	–0.60†
Cancer patient experience of treatments	0-10	3.6	1.9-6.9	8.1	1.9-15.6	13.0	5.4†	.76	0.47	0.171-0.660	0.22	0.002-0.581	0.16*	–0.43†

^*P < .05.

^†P < .01.

^‡P = .02.

The Cronbach's α exceeded .75 for all three problem scores (column 6 of Table 3). In addition, with the exception of the first problem score, factor analysis suggested a unitary solution. The first problem score had a second factor with an eigenvalue of 2.9 composed of the five questionnaire items that focused on communication about prognosis. Finally, the criterion validity was tested with two measures of the rating of the quality of care previously described (results are presented in the last two columns of Table 3). As predicted, the three problem scores showed a moderate correlation with both of these overall measures.

As a test of convergent validity, we examined the association of a higher problem score with emotional distress as measured by MHI-5. For each of the proposed problem scores, persons with a higher rate of concerns reported more distress as measured by the MHI-5. We also examined the ability of the three problem scores to differentiate between two types of practices (comprehensive cancer center v community-based oncologists). There was a trend for more reported concerns with each problem score among cancer patients receiving their care at comprehensive cancer centers compared with community-based physicians. There was no association (all P > .10) found between sex, race, or the level of education and any of the problem scores. However, there was a trend for older persons to voice fewer concerns than younger persons.

DISCUSSION

The goal of this study was to develop and validate a survey tool to measure quality of care concerns of patients with advanced-stage cancer at the time of diagnosis and beginning of treatment. Typical measures of consumer perceptions of the quality of care do not address the specific needs of this group and do not use newer survey methodologies to provide more actionable information to guide quality improvement efforts. Many of the enclosed items are important for all cancer patients at the time of diagnosis and treatment. This preliminary validation provides evidence of reliability and validity that warrants further testing to develop a measurement strategy that ensures that the patient's voice informs the assessment of the quality of cancer care.

Previous surveys that examined the cancer patient's perceptions of the quality of care did not use newer survey methodologies and/or did not addressed the unique needs of this population. These surveys focused on information needs^21–24 or particular types of cancer²⁵ or used only rating scales.²⁶ Only the Ambulatory Oncology Patient Satisfaction Survey^27,28 used newer question response tasks, as patient-centered reports, to examine quality of cancer care among all cancer patients. Unlike the Ambulatory Oncology Patient Satisfaction Survey, we chose to develop a survey that focused on the needs of cancer patients with advanced disease at two critical time points in the disease trajectory—the experience of being diagnosed with an advanced stage of cancer and beginning treatment for that stage.

Previous research highlighted the importance of communication and providing the cancer patient with adequate information about his or her prognosis and the treatment plan. The majority of respondents expressed one or more concerns with communication at the time of diagnosis or ongoing treatment. In contrast, only one third of respondents voiced any concerns about the experience of cancer treatment. However, this still represents a tangible target for improvement. Although Sandov et al²⁷ found a higher rate of concerns using patient-centered reports from a more generalizable sample of ambulatory cancer patients in Canada, our sample with lower rates of concern was based on oncologist offices that volunteered to participate in this study.

This preliminary study provides evidence of internal consistency and validity of the proposed three problem scores. Our findings further support reports that persons with more unmet needs for information experience a higher rating of psychological distress compared with patients with fewer unmet information needs.²⁴ Specifically, we found that persons who reported a higher rate of concerns with the quality of care reported a higher level of psychological distress as measured by the MHI-5.

There are important limitations to bear in mind when interpreting this study. Because of a competing protocol at one of the recruitment sites, many patients refused participation in this study at that site. On the basis of available demographic information, women were slightly more likely to participate in this study, and only a small number of minority patients participated. Focus groups were conducted with African Americans in the development of the survey to ensure that their perspective was included in the development of the instrument, but recruitment of patients for the cohort study at potential locations was not possible. The examination of the association of race and other demographic characteristics with the proposed problem scores is preliminary and limited by the number of patients and sites that participated in the study. Thus, the finding that cancer patient perceptions differed between a comprehensive cancer center and private office should be replicated among a larger sample of sites and patients. However, these preliminary results do provide evidence that warrants the further testing of this instrument in more generalizable samples and the translation of the instrument into Hispanic languages for further validation.

High-quality cancer care should address both the curing and caring aspects of care.²⁹ At times, health care providers and patients with advanced cancer that implies a life-limiting prognosis are faced with, “Hoping for the best, but preparing for the worst.”³⁰ Especially at this time, medical care must be patient centered, understanding the needs of the patients and their families, as well as their goals of care. To achieve this vision, we need surveys that collect reliable and valid data about consumer perceptions as one of the measures of the quality of cancer care. This study introduces a set of three problem scores that shows promise. Further studies will help solidify the usefulness of these measures.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The author(s) indicated no potential conflicts of interest.

AUTHOR CONTRIBUTIONS

Conception and design: Joan M. Teno, Julie C. Lima, Kathleen Doyle Lyons

Manuscript writing: Joan M. Teno, Julie C. Lima, Kathleen Doyle Lyons