CORR Insights®: Is It Possible to Develop a Patient-reported Experience Measure With Lower Ceiling Effect?

Where Are We Now?

Growing up in the Netherlands, I was taught never to strive for a perfect score. As a country whose culture is inculcated in Calvinist values, the Dutch believe that a 10 out of 10 is reserved for God, and perfect scores will not be achieved by man. In high school, scoring a 6 out of 10 was seen as “adequate,” and scoring an 8 out of 10 would put you in the top 1% of students, allowing you to graduate with honors. I find it telling that when one of my close friends who works for an American company sent out his annual satisfaction survey, he had to remind each of his Dutch customers that his bosses were American and would see anything less than a perfect score as a failure and might jeopardize his much-coveted promotion.

After medical school, I first moved to the United States to volunteer my time as a research student. I was amazed by the cultural differences between our two countries when it came to grades: I learned that most of my peers had been “straight-A students” for most of their lives, and having a perfect GPA was important to be considered for medical school. In this culture where perfect scores are the norm, patients seemed to feel pressure to give their surgeon perfect scores when we tried to measure patient satisfaction with office visits. We frequently had to dichotomize satisfaction into “perfect” and “not perfect,” as these strong ceiling effects impeded us from finding meaningful associations [6, 10]. Flash-forward several years, today I find myself in an orthopaedic residency here in the US. Having adopted a culture that strives for perfection, I now find myself disappointed when I receive an evaluation that is anything but perfect. But are we missing nuance when we use a grading system that routinely allows people to achieve perfect scores? And what can we learn from our patients if we give them space to tell us we did something other than a “perfect” job?

In this unique and intriguing study by Brinkman et al. [4], published in this issue of Clinical Orthopaedics and Related Research®, the authors used a combination of factors analysis and item response theory to generate a new patient-reported experience measure with fewer ceiling effects, approximating a more Gaussian distribution. This novel measure will allow us to document patient satisfaction with much more nuance than traditional experience measures such as the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, which has strong ceiling effects. This work is an important step forward, as prior research on patient-reported experience has been limited by a lack of detail among patients who score in the upper regions of the satisfaction measures.

Where Do We Need To Go?

Reflecting on my (early) career in orthopaedic surgery, I have come to realize that oftentimes the cases or patient interactions that did not go as well as they could have point us to important opportunities for growth. By utilizing clinical outcome or experience measures that have strong ceiling (or floor effects), we are missing out on a learning opportunity. I believe that we need to move toward utilizing instruments that score on a continuum and allow for greater degrees of nuance. One such approach involves the Patient-Reported Outcomes Measurement Information System (PROMIS) scales [8, 13], which are developed utilizing item response theory—like the current study by Brinkman et al. [4]—and are normed to a general population, which allows age-matched comparisons of patient outcomes between cohorts of patients with and without some condition of interest.

This is not just important for research. It has clinical implications, too. When we speak with patients about the expected outcomes after a surgical intervention, we often use percentages [5], such as “80% of patients who get a total knee replacement are satisfied.” It is important to realize that this subset of 20% of patients may report categorical dissatisfaction with the procedure, but that the feelings of the other 80% may fall somewhere between “I am doing great!” and “I am doing a bit better, but my knee is far from perfect.” When we use such brutally dichotomous (binary) categories for a nuanced domain like satisfaction, we obscure some important detail that might help patients in their decision-making. And a notable number of orthopaedic procedures (such as joint replacements and fusions in particular) may be considered salvage procedures; setting “perfect satisfaction” as an expectation in that context will lead to especially disappointing results for most patients.

How Do We Get There?

In order for us to gain the amount of nuance needed for our specialty to continue improving, we need to develop more measures with improved psychometric properties and lower ceiling and floor effects. Greater sensitivity in the instruments used to measure satisfaction or symptom intensity will better detect differences between procedures we perform or improvements in the care that we deliver. Although it may be appealing for a hospital system operating in a competitive market to report that “95% of our patients are satisfied,” the questionnaires that result in this kind of reporting artificially inflate the scores, and so they are not helpful tools for clinicians. My sense is that prior studies that failed to identify associations with patient satisfaction using these types of “binary” instruments would have vastly different results if repeated with more sensitive measures [3, 9, 11]. Development of improved instruments is certainly feasible, and I believe that the current study of Brinkman et al. [4] is a good illustration of the power of item response theory to develop measures with a more Gaussian distribution. Their study presents a high-quality measure for more generalized satisfaction, but future studies are indicated to develop more disease-specific instruments.

The development of these more-nuanced instruments will not only help us in a research setting but will also help us have more informative conversations with our patients. Quoting patients percentages about satisfaction using binary endpoints (yes/no) creates a false perception that success or failure of a treatment is black or white, rather than shades of gray, and may reinforce an unhealthy mindset [12].

As surgeons, we should have the humility to accept that our results will not be perfect, and we should strive to improve how we study our interventions while avoiding measures that artificially inflate our scores. Perhaps there is something to be learned from the Dutch grading system after all.

Read This Next

• To put the current study into context, I suggest having a look at the authors’ prior work on this subject, in which they study ceiling effects in satisfaction measures [14], and their early pilot work attempting to increase variation in top satisfaction scores [1].

• As I think it is important to move away from binary measures of success, I would recommend reading this column by Bernstein et al. [2] that proposes a tiered system to evaluate the success of the procedures we perform.

• Take a look at this manuscript that helps us understand factors that may improve patient satisfaction after an office visit [7], which underlines the importance of effective communication strategies.