A Qualitative Analysis of Hospital Leaders’ Opinions About Publicly Reported Measures of Health Care Quality

Article-at-a-Glance

Background

Hospital leaders play an important role in the success of quality improvement (QI) initiatives, yet little is known about how leaders engaged in QI currently view quality performance measures. In a follow-up to a quantitative study conducted in 2012, a study employing qualitative content analysis was conducted to (1) describe leaders’ opinions about the quality measures reported on the Centers for Medicare & Medicaid Services (CMS) Hospital Compare website, (2) to generate hypotheses about barriers/facilitators to improving hospitals’ performance, and (3) to elicit recommendations about how to improve publicly reported quality measures.

Methods

The opinions of leaders from a stratified sample of 630 hospitals across the United States regarding quality measures were assessed with an open-ended prompt that was part of a 21-item questionnaire about quality measures publicly reported by CMS. Their responses were qualitatively analyzed in an iterative process, resulting in the identification of the presence and frequency of major themes and subthemes.

Results

Participants from 131 (21%) of the 630 hospitals surveyed replied to the open-ended prompt; 15% were from hospitals with higher-than-average performance scores, and 52% were from hospitals with lower-than-average scores. Major themes included (1) concerns regarding quality measurement (measure validity, importance, and fairness) and/ or public reporting; 76%); (2) positive views of quality measurement (stimulate improvement, focus efforts; 13%); and (3) recommendations for improving quality measurement.

Conclusions

Among hospital leaders responding to an open-ended survey prompt, some supported the concept of measuring quality, but the majority criticized the validity and utility of current quality measures. Although quality measures are frequently being reevaluated and new measures developed, the ability of such measures to stimulate improvement may be limited without greater buy-in from hospital leaders.

The Centers for Medicare & Medicaid Services (CMS) began publicly reporting hospital performance on a core set of 10 evidence-based quality measures on the Hospital Compare website in 2005.¹ Since then, the number of measures has grown rapidly and now includes not only measures of mortality and re-admission but also measures pertaining to complications of care, patient experience, cost, and volume. By publicly reporting hospitals’ performance on these measures, CMS aims to improve patient outcomes through two mechanisms: (1) stimulating local quality improvement (QI) efforts and (2) encouraging consumers to choose higher-quality hospitals.¹ However, it remains unclear what impact public reporting has had on health care processes and patient outcomes, with the majority of studies finding less positive impact than anticipated.^2–6 The reasons for this are likely multifactorial, but one contributor may be hospital leaders’ beliefs about the validity and utility of these measures.^7–10

Successful efforts to implement organizational change generally require substantial buy-in from leadership.^11–13 Hospital leaders’ attitudes toward specific quality measures and public reporting have been quantitatively assessed previously,⁸ but we know of no current qualitative assessment of attitudes and beliefs specific to CMS measures and public reporting of performance on these measures.

To better understand leaders’ perspectives, in 2012 we conducted a survey that was intended to update and expand understanding of hospital leaders’ opinions about publicly reported quality measures.¹⁴ The survey included an open-ended prompt that allowed respondents to expand on opinions elicited earlier in the survey and to express opinions that may not have been captured in the quantitative portion of the study. In this article, we report the results of the qualitative analysis of responses to this prompt.

Methods

Questionnaire Development and Content

As previously reported, we sent a 21-item questionnaire (Appendix 1, available in online article) to a national sample of hospital leaders who were engaged in QI; it included an open-ended prompt intended to elicit respondents’ opinions about quality measures: “Please share your additional thoughts about publicly reported measures of healthcare quality, including strengths or weaknesses of current measures, ideas for new measures, etc.” The study protocol was approved by the Institutional Review Board at Baystate Medical Center.

Study Sample and Questionnaire Administration

We sampled hospitals at three levels of quality performance on the basis of their relative scores on mortality and readmission measures for pneumonia, heart failure, and acute myocardial infarction, as reported on Hospital Compare. We used CMS risk-standardized performance scores to rate each hospital as “better-than-expected,” “at-expected,” or “below-expected” performance. From the 4,459 hospitals in the Hospital Compare database, we excluded 624 (14%) because of missing data for a performance measure (resulting from low-case volumes that did not meet the CMS threshold for reporting) and 136 (3%) that had mixed scores (above expected scores for one measure but below expected for another). Sampling was driven by power calculations for the overall study, with 210 hospitals randomly selected from each performance stratum (N = 630). Hospital characteristics, such as teaching status, were obtained from matched American Hospital Association (AHA) survey data when available.

We identified the names, addresses, telephone numbers, and e-mail addresses, when available, of the CEOs and the leaders responsible for quality at selected hospitals (for example, Chief Quality Officer, Medical Director of Quality, Vice President of Medical Affairs, Chief Medical Officer) by searching the hospitals’ websites and through telephone inquiries. Two weeks before mailing or e-mailing the questionnaire, we sent a postcard or e-mail message notifying potential participants about the goals of the study. After the initial questionnaire mailing, we sent up to three reminders to hospitals that had not yet responded, and made up to three attempts to contact remaining non-respondents by telephone. A $2 bill was included with the mailed questionnaires. All responses to the open-ended prompt were included in the current analysis.

Analysis

Overview

The goal of directed qualitative content analysis is typically to extend and potentially enrich existing knowledge.¹⁵ This approach differs from conventional qualitative content analysis in that a priori codes are often developed on the basis of existing research and knowledge, with new codes added when data do not fit the existing coding scheme.¹⁵ Codes are then organized into categories and subcategories, presenting evidence of support or lack of support for theories using codes with exemplars and descriptive evidence.¹⁵ In this study, we used directed qualitative content analysis to assess respondents’ attitudes regarding quality measures and public reporting of performance on quality measures.

Preparation of Responses and Directed Qualitative Analysis

Each participant’s response was parsed into blocks of text that represented unique thoughts or concepts. We then developed an a priori provisional codebook on the basis of previous studies related to public reporting of heath care quality and review of several of the responses.¹⁶ In developing a priori codes, we also considered how the first portion of the questionnaire might have prompted respondents to discuss related topics. Responses from 10 randomly selected participants were then independently coded by three investigators [K.L.H., T.A.J., P.J.T.] trained in qualitative analytic techniques by an internist-pediatrician with expertise in qualitative methods [S.L.G.]. The codes were then discussed and discrepancies resolved through discussion among the three coders. The codebook was amended in an iterative process to incorporate new concepts as they were identified. This process was repeated until no new codes were applied. The three primary coders then independently coded the remaining blocks of text. If new concepts were identified during this stage of coding, the three coders reconvened and discussed the potential new code, with the decision to add a code made by consensus. The first 200 blocks of text were coded by all three coders to check intercoder agreement and reproducibility of the code book.^15–17 Two authors [S.L.G., T.L.] then performed second-level coding, in which codes were organized into major pertinent themes.¹⁶ We reviewed the coded text after applying secondary codes to check for accuracy of coding and theme assignment as well as completeness of second-level coding. We used descriptive statistics to assess the frequency of select codes and themes. We also characterized each respondent’s comment as positive, negative, both positive and negative, or neutral and used descriptive statistics to determine whether certain hospital and respondent characteristics were associated with positive or negative responses.

Results

Responding Hospitals and Leaders

Of the 630 hospitals surveyed, 131 (21%) provided responses to the open-ended prompt. Fifteen of the hospitals contributed responses from two leaders, for a total of 146 respondents. Of these, 27% were CEOs, 17% were Chief Quality Officers or Vice Presidents of Quality, and 19% were Chief Medical Officers. Fifty-eight percent of the respondents had either an MD or a DO degree with or without an additional master’s degree; the rest had nursing or other non-MD/DO degrees. Fifteen percent of the respondents were from hospitals rated as “better-than-expected” on mortality or readmission measures, 34% were from hospitals with “expected” performance, and 52% were from hospitals with “worse-than-expected” performance. Among the responding hospitals with AHA survey data, 42% had fewer than 200 beds, 34% were teaching institutions, and 52% were located in urban areas. Hospitals that responded to the open-ended prompt did not differ in teaching status but were more likely to be larger, rural, and score lower than expected on quality measures compared to hospitals that did not respond to the survey or did not provide a response to the open-ended prompt (Table 1, above).

Table 1.

Characteristics of Hospitals Surveyed Compared to Those That Responded to the Open-Ended Prompt

	Total N (%)	No Response to Prompt* n (%)	Response to Prompt n (%)	P Value†
Teaching‡				.80
Yes	215 (35)	171 (35)	44 (34)
No	408 (65)	321 (65)	87 (66)
No. of Beds‡				.005
< 200	332 (53)	277 (56)	55 (42)
200–400	179 (29)	138 (28)	41 (32)
> 400	110 (18)	76 (15)	34 (26)
Population Served‡				.001
Rural	202 (32)	142 (28)	60 (48)
Urban	421 (68)	355 (72)	66 (52)
Sampling Strata				.001
> expected	210 (33.3)	191 (38)	19 (15)
expected	210 (33.3)	166 (33)	44 (34)
< expected	210 (33.3)	142 (28)	68 (52)

^*Includes hospitals that did not return a questionnaire and those that returned a questionnaire but did not respond to the open-ended prompt.

^†p value (chi-square test) compares hospitals with “No response” and “Response” to prompt.

^‡Missing data—American Hospital Association data could not be matched to some hospitals.

Eighty-five percent of respondents’ comments contained a negative component. Respondents from hospitals that scored below expected on quality measures were no more likely to have provided a negative comment than those with expected or better than expected scores (chi-square test; p = .31), and leaders with an MD/ DO degree were no more likely to provide a negative comment compared to respondents with non MD/DO degrees (p = .71).

Participants’ Responses

The open-ended responses provided by the 146 participants yielded 493 discrete blocks of coded text. Codes applied to these blocks of text were unified into three major themes during second-level coding: (1) concerns regarding quality measurement, (2) positive views of quality measurement, and (3) recommendations for improving quality measurement. Each of these major themes contained subthemes (italics), which are described in detail below with illustrative quotes. Details of the line codes that contributed to each major theme and additional quotes are provided in Table 2 (pages 172–173). Agreement among coders on the 200 blocks of text coded was 91%.

Table 2.

Major Themes, Subthemes, and Associated Quotes^*

Theme: Concerns Regarding Quality Measurement
Subtheme: Validity of the Measures
… the AMI [acute myocardial infarction] mortality measure does not risk-adjust for the most important factor influencing mortality: the time from onset of symptoms to the time the patient presents for treatment. Efforts [to measure care quality] continue to objectify subjective data … it is an uncertain area at best! Decision making should be firmly placed in the realm of the physician at the patient’s bedside and not some policy manual written by a bureaucrat. The metrics are basic—and need [to be] improved. Measures do not equal up to date EBM [evidence-based medicine]. The information now being reported is 1–2 years old when seen by the public. I’m fine with public reporting, but we need metrics that are fair and meaningful! Data that accurately refects clinical quality. My fear is that the number of metrics is now clearly out of hand. We believe we are now reporting about 200 metrics to 19 different organizations (required or otherwise). I believe the process measures and the patient satisfaction measures are good to share, however without true risk adjustment and analysis it is just data and not information. Any risk-adjustment methodology should be readily available and understood by the measured. Volumes are so low that traditional statistical measures are difficult to use. While every patient’s care should be the best able to be delivered … some of the rates appear to be affected by randomness. [Measures] should be reliable, truly drive to improve outcomes, not just fill a political agenda.
Subtheme: Relevance of the Measures
“Not everything that can be counted counts, and not everything that counts can be counted.”—Albert Einstein† Restricting analysis to easily captured metrics markedly reduces the value of the measure. There is a need to accelerate the shift in focus from process of care measures to clinical outcomes. Process measures are primarily measures of documentation. I am unaware of any peer-reviewed study which has demonstrated any correlation with outcomes or improved outcomes. … [measuring and reporting quality] distracts attention from the basic reason for the patient’s presentation and care. Hospitals are managing to the data, not necessarily focusing on what may be more significant issues in their community.
Subtheme: Fairness of the Measures
Until these providers are directly impacted financially and to the same degree as the hospital for core measure and patient experience data, no hospital with medical staff that is largely independent should be financially penalized. Doctors, nursing homes, home health agencies, hospice, and other providers are much more impactful on the rates of mortality/readmission than the hospital, but it is clearly convenient to place the burden on the hospital. Too many of the increasing number of process measures are easy to be gamed. The differences between top performers and average performers for many of the measures are meaningless. For core measures and patient experience, any hospital which is strapped for money is forced into the small sampling methodology. This creates … variance if the independent practitioner is not [performing at] 100%. None of the measures take into effect socioeconomic factors. I believe that variation in mortality and readmissions is NOT random but is related to factors beyond the walls of the hospital. Demographics, availability of home care, hospice, nursing home beds, local economy, etc. CMS [Centers for Medicare & Medicaid Services] does not risk-adjust on community factors. Only patient specific factors related to comorbidity. Not robust enough. Noncompliance is a huge variable in the region. We use the IHI [Institute for Healthcare Improvement] best practices, but if the patient isn’t going to follow the plan then the hospital cannot be responsible. Some hospitals have more challenging populations to serve or have a disproportionate level of poor and/or elderly patients. This can lead to lower compliance by the patient population and therefore poorer outcomes for things like readmission rates. The truth is, for instance, that both 30-day mortality and readmission have relatively little to do with care received in the hospital (in most cases) and everything to do with what happens to the patient when they leave. I believe that variation in mortality and readmissions is NOT random but is related to factors beyond the walls of the hospital. Demographics, availability of home care, hospice, nursing home beds, local economy, etc. CMS does not risk-adjust on community factors, [only] patient specific factors related to comorbidity. Not robust enough. Current CMS core measures are unnecessarily complicated which adds to cost. Unrealistic expectations defined by CMS for hospitals assume that we will be able to change behavioral attitudes and therefore, decrease mortality rates, readmission rates, etc. The burden of measuring and monitoring are becoming more and more untenable. It would be nice to develop a value analysis that looks at the marginal improvements anticipated vs. the actual costs of implementation and “holding the gains.” Many times these feel like “unfunded mandates.” Publicly reported measures are the only way hospitals can compare to each other, but I worry about the honesty of our competitors, i.e., gaming the system, while we scrupulously report the facts.
Subtheme: Concerns About Public Reporting
The only “public” looking at the measures are other hospitals. The measures need to be meaningful, fair, and reported in a timely manner. The information now being reported is 1–2 years old when seen by the public. I think that many indicators can be misleading and [that] the general public [does] not understand how many factors can influence these statistics. Average person does not understand public report measures. I support public reporting, but until CMS and others are more honest with the public about the limitations of the reporting, they have lost a lot of credibility with me.
Theme: Positive Views of Quality Measurement
In general, I believe public reporting does cause hospitals to focus on quality improvement. Standardization of processes can and does lead to improved outcomes and lower cost. … the transparency that we are trying to create is good … I believe the advent of pay for performance … has brought the needed attention to patient care issues. The measures allow us to focus our activities around specific disease categories, procedures and/or surgical interventions. Patient care experience measures have provided a forum to differentiate health care providers.
Theme: Recommendations for Improving Quality Measurement
I think CMS should consider affecting physician compensation in relation to some of the measures, not just hospital compensation. To effectively do this we need to create measurements that are extractable electronically so we can afford to be more and more transparent. Standards should be based on clinical scientific data from medical records/medical research. Not from extrapolated MedPar‡ data. Should be reviewed and “blessed” by professional organizations. Educate the public—start earlier—schools, church, employers.

^*Quotes are provided verbatim.

^†Brainy Quote.^® Accessed Feb 26, 2015. http://www.brainyquote.com/quotes/quotes/a/alberteins100201.html.

^‡MedPAR: See Centers for Medicare & Medicaid Services. MEDPAR. (Updated: Dec 17, 2014.) Accessed Feb 26, 2015. http://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/MedicareFeeforSvcPartsAB/MEDPAR.html.

Concerns Regarding Quality Measurement

The majority (n = 373; 76%) of the 493 discrete blocks of text reflected concerns about quality measures and/or public reporting of performance. These included concerns about the validity, importance, and fairness of the measures, as well as comments about public reporting.

Concerns related to the validity of the measures included comments about perceived issues with risk adjustment and the lack of evidence for an association between process measures and outcomes, such as in the following comment:

The current focus on process measures for MI [myocardial infarction], CHF [congestive heart failure], and pneumonia do not appear to have led to measurable declines in mortality and/or morbidity and should be reconsidered.

Concerns related to the importance of the measures focused largely on the perception that measures are selected based on ease of measurement rather than actual importance as an indicator of care quality:

Our first forays into measurement focused on phenomena that are easy to measure. That doesn’t always equate to importance.

Concern was expressed that the domains measured have little to do with patient care and outcomes. For example, some respondents did not feel patient experience was a useful quality measure:

Too many people—at CMS and in the community—place an undue emphasis on the “prettiness of the facility” and the willingness or ability of the facility to spend time, energy, and money on the “fluff” rather than on patient care. Good customer service does not equal quality care.

Perceived unfairness of the measures was reported by respondents and included concerns about the potential for “gaming” of the system:

I think care is marginally improved in many (though not all) areas as a result of the public reporting requirement. Clearly “gaming” is the fastest, most effective way to improve performance.”

In a concern closely related to gaming, some respondents believed that performance is more related to a hospital’s available resources for documentation than actual care:

I think since there is so much variability in the robustness of hospital clinical documentation programs, that has an impact on comparative mortality data. The risk-adjusted mortality and readmission indicators may actually be reflecting the quality of documentation, comorbid condition documentation, and coding practices versus mortality.

Inability to change patient mix and factors that impact outcomes but are outside of the hospital’s control were also felt to be unfair.

The risk adjustment process does not take into account real and meaningful differences in patient populations for readmissions. There is a growing body of evidence regarding certain patient populations and their risk for readmissions that is being ignored while those hospitals caring for these minority populations are being compared to other hospitals caring for patients not facing the same challenges/having the same cultural values.

Criticisms of public reporting centered on the lag between when measurement is performed and when it is reported, lack of utility these reports have for patients, and a failure to clearly inform the public of the limitations of the measures reported:

I support public reporting, but until CMS and others are more honest with the public about the limitations of the reporting, they have lost a lot of credibility with me.

Additional concerns included the possibility that changes in care that are driven by economic factors, such as reduced length of stay, could have a negative effect on patient satisfaction measures:

Improved patient satisfaction is hard to achieve with … increased emphasis on reducing length of stay and utilization of resources.

Positive Views of Quality Measurement

Sixty-two (13%) of the 493 coded lines contained positive views about quality measures or public reporting of performance. Subcategories included beliefs that measuring and reporting quality can drive change, focus change efforts, and engage patients. Transparency was felt to be one aspect of quality measurement that drives change:

Our state has required [public] reporting of hospital-acquired infections. This certainly drives improvement and awareness.

Approximately half of the positive comments were general observations about the potential utility of measuring quality of care. This subcategory of statements typically started with a statement of theoretical support for quality measurement, followed by a critique of the current measures and/or public reporting:

The transparency that we are trying to create is good—we are just not yet finding the best measurements to reflect good quality of care.

Recommendations for Improving Quality Measurement

Some participants offered recommendations for how quality measurement can be improved. For example, one respondent felt that the workload associated with reporting different measures to different agencies could be reduced through coordination among agencies:

There needs to be more coordination between agencies to eliminate so much duplication.

Another respondent felt that the measures should be updated regularly to ensure ongoing pursuit of excellence:

… care measures should evolve as compliance becomes ingrained thus continuing the improvement cycle.

Another respondent recommended increasing the potential for hospitals to learn from other’s successes:

[there should be an] … opportunity to post/link what performance improvement projects [a] hospital has implemented.

Other respondents recommended expanding responsibility for quality of care beyond the hospital; one recommended developing quality measures that assess health and health care in a community.

“There is a need to develop Q [quality] measures that assess the health of a community.”

Several participants commented on the need to align physician incentives with hospital performance on quality measures:

However, CMS must align incentives with physicians and not focus solely on hospital performance. We struggle to hold the independent medical staff accountable for quality without incentives.

Discussion

In this study of hospital leaders from a diverse sample of 131 hospitals in the United States, we found that although some hospital leaders endorsed the concept of measuring quality and comparing hospitals’ performance in principle, there remain substantial concerns about the current processes for measuring and reporting quality. Criticisms of quality measures included issues related to validity, importance, and fairness of the measures. Criticisms about public reporting of hospitals’ performance included a time lag between measurement and reporting, lack of transparency about limitations of the measures used by CMS, and lack of use by consumers. Recommendations on how best to address some of the issues with quality measures and public reporting were offered.

Many of the concerns about the CMS publicly reported quality measures put forth by participants in this study are similar to those previously described by other stakeholder groups, including physicians and hospital leaders. For example, Barr et al. identified physician concerns about the methodological rigor of the measures,¹⁸ while Casalino et al. reported concerns about how public reporting of the measures would cause care to be diverted from important elements that are not measured and reported.¹⁹ Hafner et al., who used focus groups to assess hospital staff (administrators, physicians, nurses) opinions, found, similar to the current study, that these stakeholders offered some positive views but also questioned the data quality and the utility of the reports for the public.²⁰ The quantitative portion of this study¹⁴ also revealed concerns, but this study provides both a broader and more detailed description of these concerns. Other studies of hospital leaders have used only closed-ended survey questions,⁸ explored only one measure,⁷ or sought to describe QI activities rather than opinions about measures and reporting.²¹ Our study adds to this body of literature by being one of few recently conducted studies that directly elicited both CEOs’ and Quality Improvement Officers’ critiques of CMS quality measures via an open-ended inquiry. This inquiry yielded important insights as to why concerns about measures may persist, and the findings are aligned with current debates regarding use of outcome versus process measures, risk adjustment, and the roles of patient satisfaction in care quality.^22–24

Interestingly, some of the comments in the present study demonstrated a lack of awareness of both ongoing assessments of the measures’ validity and importance and availability of resources for information sharing. Specifically, one participant commented that measures should be revisited periodically to be sure they remain pertinent. This is, in fact, something that CMS does on a regular basis.²⁵ In a similar vein, one participant suggested there be a forum for hospitals to share QI strategies they have undertaken. The Agency for Healthcare Research and Quality’s (AHRQ) Health Care Innovations Exchange currently offers such a forum.²⁶ Finally, one participant expressed a belief that there was more concern about appearance than substance at CMS, counter to stated goals of the Hospital Compare reporting system. These findings suggest that more effective outreach and engagement of leaders in CMS and AHRQ efforts to ensure that the measures are valid, meaningful, and fair may reduce some of the negative perceptions expressed.

Although some of the respondents in this study voiced support for the potential value of quality measurement, the concerns they expressed about current measures may represent barriers to improvement. The importance of leadership in bringing about change in an organization is well described.^{9,11–13,21,27–39} This included engagement in quality initiatives at the level of the board and leader incentives for QI. This study suggests that there may be persistent doubts about the validity of current systems of measuring and reporting quality, likely limiting enthusiasm for initiating changes within their institutions. Efforts to bolster confidence in these measures will need to address these specific concerns.

Our findings should be interpreted in light of several limitations. First, only 21% of the participants who returned questionnaires responded to the open-ended prompt; additional perspectives may have been identified with a greater number of respondents. Second, it is possible, given the overlap between topics addressed in the quantitative portion of the questionnaire and categories/themes that emerged in the qualitative analysis, that respondents may have been primed to share opinions about the topics introduced earlier in the questionnaire. Third, this analysis is qualitative and should be reinforced by quantitative measurement of the prevalence of the opinions expressed in this survey. Finally, respondents differed from nonrespondents in several of the characteristics measured, including having lower performance on quality scores. This may mean that the findings of this study are most pertinent to those who share the characteristics of respondents.⁴⁰

Conclusion

Although quality measures used to evaluate performance on the CMS Hospital Compare website appear to enjoy theoretical support among the hospital leaders who are a target audience for these data, these leaders express ongoing doubts about the validity, importance, and fairness of the current measures. The specific concerns expressed may indicate a role for broad educational campaigns about the development and ongoing evaluation of the measures. Until hospital leaders trust that the measures are fair and meaningful, their endorsement of strategies to improve quality in their hospitals may be suboptimal because their attitudes and beliefs may strongly influence institutional action, and thus may limit what can be achieved through quality measurement and reporting.