Bias in patient experience scores in radiation oncology: a multicenter retrospective analysis

Abstract

Objective:

Patient experience scores are increasingly important in measuring quality of care and determining reimbursement from payers, including the Hospital Value Based Purchasing Program and the Radiation Oncology Model. However, the role of bias in patient experience scores in oncology is unknown, raising the possibility that such payment structures may inadvertently perpetuate bias in reimbursement. Therefore, we characterized patient-, physician-, and practice-level predictors of patient experience scores in patients undergoing radiation therapy.

Methods:

We retrospectively reviewed patient experience surveys for radiation oncology patients treated at two large multi-site academic cancer centers. The outcome was responses on four survey questions. Co-variates included self-reported patient demographics, physician characteristics, practice setting characteristics, and wait time rating linked to each survey. We fitted multivariable ordinal regression models to identify predictors of receiving a higher score on each of the survey questions.

Results:

2,868 patients completed surveys and were included in the analysis. Patient experience scores were generally high, with >90% of respondents answering 5/5 on the 4 survey items. Physician gender was not associated with any measured patient experience outcomes (all p-values>0.4). Independent predictors of higher score included a wait-time experience classified as “good” as compared to “not good” (all q-values<0.001).

Discussion:

Oncology practices aiming to improve patient experience scores may wish to focus their attention on improving wait times for patients. While we did not observe a difference in patient experience scores based on physician gender, such bias is likely to be complex and further research is needed to characterize its effects.

Summary Sentence:

Patient experience scores at radiation oncology practices are associated with modifiable characteristics of the clinic experience, such as wait times, and do not appear to be affected by demographic characteristics of physicians or patients.

INTRODUCTION:

Scores on patient experience surveys are widely used across healthcare to help measure quality and are increasingly used to help determine reimbursement to providers. Under the Patient Protection and Affordable Care Act of 2010, patient experience scores are considered in determining value-based incentive payments to hospitals[1], and in 2019, the United States Centers for Medicare & Medicaid Services (CMS) announced an episode-based payment model for radiation therapy, known as the Radiation Oncology Model (RO Model), which will use patient experience scores as a quality measure to help determine reimbursement to radiation oncology practices[2]. In the RO Model, beneficiary experience will be measured using the Consumer Assessment of Healthcare Providers and Systems (CAHPS®) Cancer Care Radiation Therapy Survey and a 2% pay-for-performance quality withhold from the professional component of the payment amount will incorporate these experience measures[2].

While there is an extensive body of literature regarding determinants of the patient experience across healthcare, studies aimed at understanding this construct within oncology in the United States are limited. As a result, oncology practices have insufficient information to make informed decisions about how to invest resources as patient experience scores continue to carry more weight in measuring quality of care. Further, while these quality-linked payment structures have the benefit of incentivizing patient-centered care, patient experience is a highly complex and subjective construct, susceptible to individuals’ biases. Therefore, the increased importance placed on patient experience scores carries the risk of perpetuating existing racial and gender disparities within the medical field. For both of these reasons, a more detailed understanding of the factors that contribute to patient experience scores within oncology is needed.

Previous work has identified a number of factors that are associated with patient experience for individuals with cancer[3–7]. Herein, we build on prior work by assessing, to our knowledge, the most comprehensive set of predictors of patient experience studied in radiation oncology. We hypothesized that non-modifiable physician demographic factors such as gender and gender concordance would significantly affect scores on patient experience surveys. To test this hypothesis, we examined the independent effects of patient, physician, and practice level factors on patient experience in radiation oncology across two large academic cancer centers. Our results will provide important information to policymakers about the need to consider possible sources of bias when relying on patient experience data, and to oncologists and practices working to adapt to policy changes that place increased importance on the patient experience.

METHODS:

Design and cohort

We performed a retrospective analysis of patient experience survey data for adult patients seen in consultation at two large cancer centers with regional clinics across two states from 2017 to 2019. Standardized patient surveys used to routinely assess patient experience were collected via text, email and paper within one month after the visit. Five survey questions regarding physician-patient communication and practice experience that were utilized on both institutions patient experience surveys were selected for the study to examine the association of patient, practice and practice related factors and patient experience scores, including: 1) likelihood to recommend the practice (“How likely are you to recommend our practice to others?”); 2) physician friendliness (“friendliness of the physician”); 3) physician concern (“concern the physician showed for your questions or concerns?”); 4) physician explanation (“explanation the physician gave you about your condition?”); and, 5) wait time (“How would you rate the waiting time during today’s visit?”). All patients who completed a patient experience survey after a consultation visit during the period were included in the analysis. Physicians were excluded if they had fewer than 50 surveys completed (N=9) to ensure reliability of physician-level estimates. Responses were excluded if the response regarding wait time was not responded to (N=13) because wait time was used as a co-variable in our model given the strong association of wait time and patient experience scores[8–11]. Finally, responses were excluded if the patient’s socioeconomic status was unknown (N=96).

Patient characteristics

We linked self-reported patient demographics to survey responses, including gender (male/female), age (<=65/>65), race (non-Hispanic white vs. non-white minority/other), socioeconomic status (SES) (high vs. low), cancer type (breast; genitourinary [prostate, bladder, kidney]; head/neck, skin and other; central nervous system, spine and bone metastases; gastrointestinal and thoracic; and, gynecologic [cervix, uterine], lymphoma and sarcoma), and receipt of radiation (treated vs. not treated). Socioeconomic status categories were derived using the patient’s street address from the Neighborhood Atlas[12], with high and low categories derived using national deciles where the top two deciles (representing approximately 75% of the sample) were classified as high SES.

Physician and practice characteristics

We also linked physician- and practice-related variables to survey responses, including gender (male/female), years of experience (<=7/>7), and clinic volume (high vs. low) [Table 2]. Average clinic volume was defined as average number of new starts and visits by the number of clinic days per week. Clinical volume and years of experience categories were derived using the population median as a cut point. Exclusion of physician race/ethnicity was required to ensure physicians would not be identifiable.

Table 2.

Physician and Practice Characteristics

Characteristic	Physicians; N=58 (%)	Male; N (%)	Female; N (%)
Gender, n (%)
Male	41 (70.7)	41 (100)	--
Female	17 (29.3)	--	17 (100)
Years in practice
<=7, n (%)	32 (55.2)	23 (56.1)	9 (52.9)
>7, n (%)	26 (44.8)	18 (43.9)	8 (47.1)
Average clinic volumea
Low, n (%)	26 (44.8)	21 (51.2)	5 (29.4)
High, n (%)	32 (55.2)	20 (48.8)	12 (70.6)

Abbreviations: SD, standard deviation; IQR, interquartile range.

^aAverage clinic volume calculated by dividing the number of average weekly new starts/visits per week by the number of clinic days per week for a given provider.

Analyses

Our primary outcome was the likelihood of receiving a higher score on the survey item rating the likelihood to recommend the practice. Secondary outcomes included the likelihood of receiving a higher score for questions about physician friendliness, physician explanation, and physician concern. Association between patient and physician characteristics was performed by using the chi-square test. In order to assess the strength of association of potential predictors of patient experience with the scores, we discretized the scores on an ordinal scale[13, 14]: lower non-good scores (scores ranging between 1 to 3), good scores (4) and very good scores (5).

Univariable analysis examining the association between experience scores and physician gender and wait time [not good (1–3) vs. good (4–5)] were calculated using ordinal regression models with a logit cumulative link function by incorporating physician level random effects. We then fitted multivariable ordinal regression models with a logit cumulative link function by incorporating physician level random effects for potential predictors of receiving a higher score on each of the survey questions, while adjusting for patient demographics, physician demographics, practice variables and wait time response [not good (1–3) vs. good (4–5)], with clustering by physician[13, 14]. Covariates in the model were chosen a priori based on hypotheses and existing data supporting their influence on patient experience scores[10, 15–18] Scores from survey questions were used as outcome variables. For each score, a patient gender and physician gender interaction term was added to the model to determine whether it improved the model warranting further exploration of gender concordance.

Two-sided p-values with an alpha level <0.05 were applied to all statistical tests, and q-values were determined for false discovery rate correction for multiple testing. Analysis was performed with RStudio version 1.1.456 (R Core Team, 2019, Vienna, Austria) and the ‘ordinal’ package[19]. This study was reviewed and approved by the Icahn School of Medicine at Mount Sinai institutional review board.

RESULTS:

We included 2,868 individual patients who completed patient experience surveys after a consultation with one of 58 individual radiation oncologists and for whom demographic data was available. Table 1 presents the characteristics of the patients included. Physician demographics are presented in Table 2. Survey responses were generally positive, with over 90% of respondents giving a top-box score on likelihood to recommend (2626/2868, 91.6%), physician friendliness (2490/2649, 94.0%), physician explanation (2619/2861, 91.5%), and physician concern (2627/2864, 91.7%). The wait time question received more mixed responses, with 1876 (65.4%) assigning a top box score, 583 (20.3%) assigning 4/5, and 409 (14.3%) assigning a score of 1–3 (Table 3).

Table 1.

Characteristics of Patient Respondents

Characteristic	Total Respondents, N=2868 (%)
Gender, n (%)
Female	1270 (44.3)
Male	1598 (55.7)
Age, years
<=65, n (%)	1523 (53.1)
>65, n (%)	1345 (46.9)
Race/Ethnicity, n (%)
Non-hispanic white	2154 (75.1)
Non-white minority/other	714 (24.9)
Socioeconomic Percentile
Higher, n (%)	2092 (72.9)
Lower, n (%)	776 (27.1)
Cancer Type, n (%)
Breast	559 (19.5)
Genitourinary	806 (28.1)
Head & Neck, Other	224 (7.8)
CNS, Brain/Bone Metastases	682 (23.8)
Lymphoma, Cervical, Uterine	187 (6.5)
Lung, Gastrointestinal	410 (14.3)
Radiation Received, n (%)
No	942 (32.8)
Yes	1926 (67.2)
Visit Site, n (%)
Main	1513 (52.8)
Satellite	1355 (47.2)
Gender Concordance, n (%)
No	1234 (43.0)
Yes	1634 (57.0)

Abbreviations: IQR, interquartile range; CNS, central nervous system. Two-sided p-values with an alpha level <0.05 were applied to all statistical tests.

^aSocioeconomic Status (SES) determined by national percentile per the Neighborhood Atlas (1=highest SES, 100=lowest SES). Higher SES represents the top 75% of the sample, with a national percentile <22. Lower SES represents the lowest 25% of the sample, with a national percentile >= 22.

^bGynecologic includes cervix and uterine

^cGastrointestinal includes esophageal, gastric, colorectal, hepatobiliary and pancreas.

Table 3.

Patient Experience survey Items and response rates among the sample

Survey Item	Total Respondents, N (%), Total N=2868	Score of 1–3 of 5, n (%)	Score of 4 of 5, n (%)	Score of 5 of 5, n (%)
Likelihood to Recommend Practice: Raw score on 5-point scale for “How likely are you to recommend our practice to others?”	2868 (100.0)	55 (1.9)	187 (6.5)	2626 (91.6)
Physician Friendliness: Raw score on 5-point scale for “How would you rate the friendliness of the physician?”	2649 (92.4)	37 (1.4)	122 (4.6)	2490 (94.0)
Physician Explanation: Raw score on 5-point scale for “How would you rate the explanation the physician gave you about your condition?”	2861 (99.8)	52 (1.8)	190 (6.6)	2619 (91.5)
Physician Concern: Raw score on 5-point scale for “How would you rate the concern the physician showed for your questions or concerns?”	2864 (99.9)	48 (1.7)	189 (6.6)	2627 (91.7)
Wait time: Raw score on 5-point scale for “How would you rate the waiting time during today’s visit?”	2868 (100)	409 (14.3)	583 (20.3)	1876 (65.4)

Responses: 1=very poor, 5=very good

On univariable analysis, physician gender was not associated with receiving higher patient experience scores for likelihood to recommend (p=0.57), physician friendliness (p=0.47), physician explanation (p=0.67), or physician concern (p=0.51) (Figure 1a). On multivariable analysis, physician gender was not found to be a significant independent predictor of patient experience scores after adjusting for other patient, physician and practice factors. For male physicians, odds of higher scores did not change on likelihood to recommend (OR 0.97 (95% CI 0.66–1.43), q>0.9), physician friendliness (OR 1.33 (0.84–2.12) q=0.8), physician explanation (1.09 (0.71–1.67), q>0.9), or physician concern (OR 1.24 (0.84–1.85), q=0.5). Other patient-level factors, including race/ethnicity, age, socioeconomic status, and cancer type did not have a significant association with the odds of receiving higher patient experience scores in any of the questions analyzed. The addition of a patient gender and physician gender interaction term did not improve the model fit for any of the survey response scores therefore further exploration of gender concordance was not pursued.

Figure 1:

Unadjusted proportion of patients assigning top box scores on each patient experience measure by physician gender (A), and by wait time (B). Good wait time is defined as a score of 4 or 5 on the wait time item, while not good wait time is defined as a score of 1–3. Physician gender was not associated with patient experience scores, while good wait time was associated with higher patient experience scores for all measures.

On univariable analysis, good wait times were associated with a higher patient experience scores for likelihood to recommend (p<0.001), physician friendliness (p<0.001), physician explanation (p<0.001), and physician concern (p<0.001)(Figure 1b). The strongest independent predictor of higher patient experience scores in multivariable analysis was rating the wait time as good, as compared to not good. Good wait time was associated with higher scores on likelihood to recommend (OR 10.06 (95% CI 7.51–13.47), q<0.001), physician friendliness (OR 6.04 (4.29–8.51), q<0.001), physician explanation (OR 4.64 (3.45–6.24), q<0.001), and physician concern (OR 5.08 (3.77–6.83), q<0.001) (Table 4). The odds of receiving higher scores was not higher for patients receiving care at satellite sites as compared to the main site after adjusting for multiple comparisons for likelihood to recommend (OR 1.71 (95% CI 1.15–2.53), q=0.056), physician friendliness (OR 1.62 (1.00–2.63), q=0.4), physician explanation (OR 1.57 (1.04–2.38), q=0.2), and physician concern (OR 1.53 (1.03–2.28), q=0.2) (Table 4).

Table 4.

Adjusted estimates for likelihood of receiving a higher score on patient experience survey response items in radiation oncology

	Likelihood to Recommend Practice		Physician Friendliness		Physician Explanation		Physician Concern
Covariable	Odds Ratio (95% CI)	Q valuee	Odds Ratio (95% CI)	Q valuee	Odds Ratio (95% CI)	Q valuee	Odds Ratio (95% CI)	Q valuee
Physician Gender
Female	reference		reference		reference		reference
Male	0.97 (0.66, 1.43)	>0.9	1.33 (0.84, 2.12)	0.8	1.09 (0.71, 1.67)	>0.9	1.24 (0.84, 1.85)	0.5
Years in practice
<=7	reference		reference		reference		reference
>7	0.97 (0.67, 1.41)	>0.9	0.88 (0.56, 1.38)	0.8	0.92 (0.61, 1.40)	>0.9	0.71 (0.48, 1.04)	0.2
Clinical Volumea
High	reference		reference		reference		reference
Low	0.71 (0.49, 1.01)	0.3	1.08 (0.68, 1.72)	>0.9	1.02 (0.68, 1.52)	>0.9	0.70 (0.49, 1.02)	0.2
Race/Ethnicity
Non-hispanic white	reference		reference		reference		reference
Non-white minority/unknown	0.99 (0.71, 1.39)	>0.9	1.08 (0.72, 1.61)	>0.9	1.01 (0.73, 1.41)	>0.9	1.05 (0.76, 1.46)	0.8
Patient Gender
Female	reference		reference		reference		reference
Male	1.15 (0.77, 1.74)	0.8	0.82 (0.51, 1.32)	0.8	1.22 (0.82, 1.82)	0.7	1.06 (0.70, 1.60)	0.8
Patient Age
<=65	reference		reference		reference		reference
>65	1.07 (0.79, 1.43)	>0.9	1.12 (0.79, 1.58)	0.8	1.00 (0.75, 1.32)	>0.9	1.19 (0.89, 1.59)	0.5
Patient Socioeconomic Statusb
Higher	reference		reference		reference		reference
Lower	0.92 (0.67, 1.28)	>0.9	0.87 (0.60, 1.28)	0.8	1.10 (0.79, 1.51)	>0.9	0.86 (0.63, 1.18)	0.5
Cancer Type
Breast	reference		reference		reference		reference
Prostate, Bladder, Kidney	1.22 (0.68, 2.20)	0.8	0.99 (0.49, 2.02)	>0.9	1.28 (0.72, 2.27)	0.7	1.09 (0.61, 1.96)	0.8
Head & Neck, Other	1.02 (0.52, 2.00)	>0.9	1.87 (0.71, 4.95)	0.8	1.08 (0.56, 2.08)	>0.9	1.11 (0.56, 2.20)	0.8
CNS, Brain/Bone Metastases	1.26 (0.77, 2.05)	0.8	1.00 (0.54, 1.85)	>0.9	1.44 (0.89, 2.31)	0.5	1.57 (0.95, 2.58)	0.2
Lymphoma, Gynecologicc	1.49 (0.73, 3.02)	0.8	1.36 (0.55, 3.36)	0.8	2.25 (1.05, 4.84)	0.2	1.49 (0.75, 2.94)	0.5
Lung, Gastrointestinald	1.57 (0.86, 2.87)	0.5	1.07 (0.52, 2.22)	>0.9	1.36 (0.77, 2.38)	0.7	1.33 (0.75, 2.38)	0.5
Radiation
Not Treated	reference		reference		reference		reference
Treated	1.13 (0.83, 1.54)	0.8	1.23 (0.86, 1.75)	0.8	1.16 (0.86, 1.56)	0.7	1.14 (0.84, 1.53)	0.6
Visit Site
Main	reference		reference		reference		reference
Satellite	1.71 (1.15, 2.53)	0.056	1.62 (1.00, 2.63)	0.4	1.57 (1.04, 2.38)	0.2	1.53 (1.03, 2.28)	0.2
Wait Time
Not Good	reference		reference		reference		reference
Good	10.06 (7.51, 13.47)	<0.001	6.04 (4.29, 8.51)	<0.001	4.64 (3.45, 6.24)	<0.001	5.08 (3.77, 6.83)	<0.001

Abbreviations: CI, confidence interval; CNS, central nervous system. Two-sided p-values with an alpha level <0.05 were applied to all statistical tests, and q-values were calculated to correct for multiple comparisons.

^aAverage clinic volume calculated by dividing the number of average weekly new starts/visits per week by the number of clinic days per week for a given provider

^bSocioeconomic Status determined by national decile per the Neighborhood Atlas.

^cGynecologic includes cervix and uterine

^dGastrointestinal includes esophageal, gastric, colorectal, hepatobiliary and pancreas.

^eFalse discovery rate correction for multiple testing

DISCUSSION:

In our analysis, non-modifiable factors such as patient-physician gender concordance and other patient and physician demographic factors did not have a significant association with patient experience scores. In contrast, wait time ratings were associated with patient experience scores for radiation oncology patients. Treatment at a satellite site rather than the main center was not a significant predictor of patient experience scores after correction for multiple hypothesis testing. Collectively, our data provide guidance for understanding potential bias in patient experience ratings assigned by cancer patients. While prior research has addressed predictors of patient experience in oncology, to our knowledge this study is the first to specifically examine the relationship between physician demographic factors and experience scores among oncology patients, carrying broad implications for the potential for bias inherent in this method of evaluating healthcare quality.

In the literature outside of oncology, research into the factors associated with outcomes on patient experience surveys has demonstrated that non-modifiable demographic characteristics can impact scores. Specifically, the gender of both the physician[15, 17, 20, 21] and the patient[22–24] have been shown in certain settings to affect scores, with female physicians receiving lower ratings, and female patients tending to give higher ratings. However the data in this area is inconsistent, with other studies showing no difference in patient experience scores based on physician or patient gender[25–29], and a meta-analysis conducted in 2011 showing a negligible effect in favor of female physicians[30]. Extensive research has focused on understanding gender disparities in how patients perceive their physicians, showing convincingly that any effect that does exist is not due to lower competence among female physicians compared to male colleagues[31–35]. Instead, a recent review of the literature proposed a conceptual framework of this phenomenon centered around patients’ gendered expectations of physicians’ care style[36].

Our results showing no significant relationship between physician or patient gender and satisfaction scores add to this body of literature. The finding of no relationship between gender and patient experience among oncology patients is reassuring, and may help to alleviate concerns about disparities in reimbursement. While there is no consensus yet about the role of gender bias in patient experience scores, the contrast between our results and certain studies showing significant disparities should be considered. One possible explanation is an inherent difference between the oncology population and the primary care or other subspecialty populations where prior research has been done, consistent with the observation that physician specialty impacts patient experience scores[17, 25]. Another possibility is that our results focused on experience ratings assigned to the practice rather than the provider, which may dilute the effects of individual bias. In the current study, we assessed initial consultation with a radiation oncologist, which most often indicates a new rather than long-standing relationship, and therefore we hypothesized the greatest risk for bias, though this was not borne out by the data.

Our analysis did demonstrate a significant association between wait time and patient experience. Prior work has convincingly shown that longer wait-times are negatively correlated with patient experience -- not only for overall satisfaction, but for perceived quality of care as well[10, 37–39]. Within radiation oncology, wait times have been shown to correlate with patient experience, with patients who report longer wait times reporting lower scores on patient experience measures[4]. However, this prior study was not able to control for patient demographic factors or satellite facility. Our results show a strong negative relationship between wait times and patient experience scores even after controlling for facility type. This observation is not surprising given the time-sensitive and highly stressful nature of radiation consultation, and may potentially be extrapolated to radiation treatment which is time-intensive. However, controlling for facility type in our analysis may not be generalizable to independent centers that are not connected a larger health system or organization that may influence patient’s perceptions of their care[40]. This finding also provides an actionable goal, as other investigators have studied the radiation oncology workflow with the goal of reducing wait time[41, 42], and thoughtful investment in clinic operations has been shown to decrease wait times for oncology patients in the infusion center setting[43]. Another potential solution to reducing wait times may be increasing use of telemedicine, which reduces travel and wait times for patients, and may reduce the burden on clinics. One study suggests high patient satisfaction with telemedicine which appears largely driven by reduced need for travel and waiting[3], providing a possible way to improve this aspect of the patient experience.

Previous research has also demonstrated that the size of the facility where treatment is delivered has important effects on patient experience scores, though the direction and magnitude of the effect appears to be mixed [44–49]. Within radiation oncology, a recent study also showed an association between higher scores and treatment at a satellite facility[50]. This observation is not surprising given the high burden of daily travel and treatment placed on radiation oncology patients. Nonetheless, our analysis did not show a statistically significant difference in patient experience scores by facility type. Therefore, despite satellite facilities providing patients the option of shorter commutes and less complicated facilities to navigate, modifying access to these facilities can be challenging and expensive, and the data regarding patients’ experience in these facilities remains inconclusive.

Our study does have several limitations. First, as outlined above, our results include questions about patient experience with a practice as a whole, rather than with individual physicians, which may dilute the effect of bias on both practice ratings and physician-specific ratings. Second, our results are not based on the same questions that are used to determine patient experience for reimbursement purposes, which may limit the generalizability of our results. However, the themes of the questions posed in the CAHPS Survey and others used by CMS are thematically similar to those assessed by the survey items in the current study, and our results offer an acceptable proxy. Our analysis was also limited to the four survey items included because the remainder of survey items were not asked at both institutions included in the study. The multi-institutional nature of our study improves generalizability of our results to other academic cancer centers, though as a result of this the effects of predictors on other patient experience outcomes were not measured. In addition, our study was also limited by our inability to include an assessment of non-responders[51]. However, our response rates provided by the survey administration providers were on average 16% and 20%, which aligns with expected response rates to CAHPS surveys. We also adjust for several patient factors and practice setting factors that account, in part, for non-response bias; however, additional patient or setting characteristics likely influence these ratings that we cannot account for[52, 53]. Despite these limitations, our results will provide useful information to policy-makers as they make decisions about how to structure payments for oncology practices, as well as administrators looking to adapt to evolving payment models that increasingly reward higher-value care.

Take Home Points:

• We show that patient experience scores at radiation oncology practices are influenced by modifiable characteristics of the clinic experience, and do not appear to be affected by demographic characteristics of physicians or patients.

• This finding is reason for optimism about policy decisions that place increased importance on patient experience.

• With the introduction of novel payment models that place increased focus on the patient experience, it is important to understand the determinants of these constructs in order to avoid inadvertently making changes that lead to biased reimbursement.

• Ultimately, bringing financial incentives into line with measures of quality and patient experience has the potential to increase the value of healthcare services, carrying benefits both for patients and for the health system as a whole.