Factors Associated With Patient Satisfaction Measured Using a Guttman-Type Scale

Abstract

Patient experience measures such as satisfaction are increasingly tracked and incentivized. Satisfaction questionnaires have notable ceiling effects that may limit learning and improvement. This study tested a Guttman-type (iterative) Satisfaction Scale (GSS) after a musculoskeletal specialty care visit in the hope that it might reduce the ceiling effect. We measured floor effects, ceiling effects, skewness, and kurtosis of GSS. We also assessed factors independently associated with GSS and the top 2 possible scores. In this cross-sectional study, 164 patients seeing an orthopedic surgeon completed questionnaires measuring (1) a demographics, (2) symptoms of depression, (3) catastrophic thinking in response to nociception, (4) heightened illness concerns, and (5) satisfaction with the visit (GSS). Bivariate and multivariable analyses sought associations of the explanatory variable with total GSS and top 2 scores of GSS. Accounting for potential confounding using multivariable analysis, lower satisfaction was independently associated with greater symptoms of depression (β: −0.03; 95% CI: −0.05 to −0.00; P = .047). The top 2 scores of the GSS were independently associated with women (compared to men: odds ratio [OR]: 2.12, 99% CI: 1.01-4.45, P = .046) and lower level of education (masters’ degree compared to high school; OR: 0.16, 95% CI: 004-0.61, P = .007). The GSS had no floor effect, a ceiling effect of 38%, a skewness of −0.08, and a kurtosis of 1.3.

The 38% ceiling effect of the iterative (Guttman-style) satisfaction measure is lower than ordinal satisfaction scales, but still undesirably high. Alternative approaches for reducing the ceiling effect of patient experience measures are needed.

Introduction

Patient experience measures such as satisfaction are increasingly obtained and incentivized (1 -9). Patients who rate their care as satisfying are more likely to adhere to clinician recommendations, keep office appointments, and refrain from formal complaints and lawsuits (1,10,11). Satisfaction is a multidimensional and complex concept that represents a distillation of perceptions and values (what matters most) (1,2,12). Provision of high-quality clinical services alone is not sufficient for patient satisfaction (1). And satisfaction does not consistently or strongly correspond with patient reported outcomes (symptom intensity and magnitude of limitations), objective physical outcomes (eg, an technically adequate total knee replacement), or mental health (3,10,13). For example, magnitude of limitation is associated with patient satisfaction with treatment of hand problems (1,14,15), but not spine problems (3). Satisfaction correlates best with other patient experience measures such as perceived clinician empathy and communication effectiveness, and there is mounting evidence that they measure a single underlying construct that might be considered as “relationship” (1,2,16).

All of the patient experience measures (satisfaction in particular) have notable ceiling effects: in most studies and in clinical practice, more than half of patients give a top score. This makes it difficult to learn from grades of satisfaction among relatively satisfied patients and to evolve and improve patient care (10,13,17 -19). Orthopedic surgeons have the lowest average outpatient satisfaction rating, among 28 specialties (20). It would be helpful to study the relative influence of structure (location, parking, waiting room), customer service, disease state, mental health, social health, communication effectiveness, and other factors on satisfaction with an office visit or with overall care. Satisfaction measures that incorporate many of these features in relatively long questionnaires make assumptions about the factors that embody satisfaction, and they are associated with lower response rates (21). It would be ideal to measure satisfaction independent from other factors, using a few questions or, if possible, just one or a few questions with limited ceiling effect.

Guttman-style questionnaires are iterative and branching. Each question in the sequence depends on the answer to the last question. They were developed as an alternative to the more common Likert scaling (22). We hoped that the branching might lead to a more normal distribution of scores. The scale is most commonly used when there is a need to use a short questionnaire with good discriminant ability (22). This study tested a Guttman-type (iterative) Satisfaction Scale (GSS) after a musculoskeletal specialty care visit in the hope that it might reduce the ceiling effect. We also measured floor effects, ceiling effects, skewness, and kurtosis of GSS, and we assessed factors independently associated with GSS and the top 2 possible scores.

Materials and Methods

Study Design

A total of 164 patients seeing one of 10 orthopedic surgeons in several different offices between June and July 2019 were invited to participate under a protocol approved by the human research committee. A research assistant not involved in patient care recruited patients directly after their visit in the office. Inclusion criteria were both new and return patients, aged between 18 and 89 years old, and English fluency and literacy. We were granted a waiver of documentation of informed consent: completion of the questionnaires implied consent.

Outcome Measures

Patients were asked to complete demographics and questionnaires measuring depression, pain catastrophizing, health anxiety, and satisfaction directly after the visit. The surgeon was asked to provide the diagnosis. All questionnaires were administrated on an encrypted tablet via secure, Health Insurance Portability and Accountability Act-compliant electronic platform: REDCap (Research Electronic Data Capture), a secure web-based application for building and managing online surveys and databases (23).

The demographics questionnaire consisted of age, gender, new or return patient, level of education, marital status, race/ethnicity, occupation, income, and insurance status.

Depression was measured using the Patient Reported Outcome Measurement Information System Depression Computer Adaptive Test (24). The scale asked statements about the last 7 days and uses a 5-point Likert scale from never to always. The total score is scaled between 0 and 100 and is reported as a T-score metric, with 50 representing the population mean and each 10 points away from the mean representing a standard deviation, with higher scores indicating more symptoms of depression (24,25).

The 4 question version of the Pain Catastrophizing Scale (PCS-4) was used to measure catastrophic thinking, a less effective cognitive coping strategy based on misinterpretation of nociception (26). The PCS-4 contains 4 questions, answered on a 4-point Likert scale, from “0- Not at all” to “4- All the time.” The total score ranges from 0 to 16 with higher scores indicating more catastrophic thinking.

The Shorts Health Anxiety Inventory (SHAI) measures cognitive factors associated with hypochondriasis. The questionnaire contains 3 factors that assess the perceived likelihood and severity of becoming ill and body vigilance (27,28). We used a 5-question version (SHAI-5) with questions ranging from 0 to 3, with a total score between 0 and 15 (28).

Satisfaction was measured with a GSS (Supplemental Appendix 1). This is just a different rating methods compared to a Likert, visual analogue, or ordinal scale. There is validation of Guttman-type scales, but to our knowledge, they have not been used to measure satisfaction. Guttman scales are iterative. The answer to the first question determines the answer to the next. Each question determines a more negative or more positive stance. The first statement in our GSS is “Today’s visit was satisfactory.” People who answer no were given progressively more negative statements to agree or disagree with. People who answered yes were asked to agree or disagree with more positive statements. The most positive statement we designed to be so superlative that few people would agree with is: “Today’s visit was more satisfying than I could imagined.” The final score obtained for the Guttman scale is determined by the most negative to most positive item the patient has agreed with (22). The total score is between 0 and 7, with higher scores indicating greater satisfaction.

Study Population

A total of 164 patients completed the questionnaires, 9 patients were excluded from the analysis because they left before completing a substantial portion of the questionnaires. The mean age (and standard deviation) of the remaining 155 patients was 51 ± 17 years old (range: 18-87 year, Supplemental Appendix 2); 72 (48%) patients were new patients and 51% of the patients visited the surgeon because of a traumatic issue.

Data Analysis

An a priori power analysis indicated that a sample of 161 patients would provide 90% statistical power to detect, with α at 0.05, for a linear regression model with 5 variables if any factor would account for 5% or more of the variability in satisfaction, and the complete model would account for 20% of the overall variability. However, after exclusions, we had 155 patients. A post hoc power analysis indicated that a sample of 155 patients would provide 90% statistical power.

The distributions of continuous variables and assumptions concerning non-normality were assessed to determine the appropriateness of the statistical tests. Descriptive statistics are presented as the mean ± standard deviation for continuous variables and proportions for discrete variables. Bivariate analyses were conducted to test the association of the GSS with each explanatory variable. We used Spearman’s rank correlation coefficient for continuous variables, Mann-Whitney for dichotomous variables, and Kruskal-Wallis test for categorical variables. We considered P < .05 as significant. We also included variables with P < .10 in a multivariable linear regression model with the GSS.

We calculated the floor- and ceiling effect and the skewness and kurtosis of the GSS. We used the highest score for the ceiling effect and the lowest score for floor effect. Skewness and kurtosis are rough indicators for a normal distribution of values. Symmetric distributions have a skewness of 0 and a kurtosis of 3 (9,18).

Also, bivariate analysis was conducted to test factors associated with the top 2 possible scores of GSS. We used t test for continuous variables and chi-square for dichotomous and categorical variables. Variables with P < .10 were included in a logistic regression model with the GSS.

Results

Factors Associated With GSS

In bivariate analysis, factors associated with a higher satisfaction score included older age (ρ: −0.03, P = .047), women (5.8 ± 1.3, men 5.3 ± 1.3, P = .006), less education (high school or less: 6.3 ± 1.1, some college: 5.6 ± 1.3, college graduate: 5.5 ± 1.3, and master’s degree or more: 5.1 ± 1.1; P = .002), not working (working: 5.4 ± 1.3, retired: 5.9 ± 1.3, unemployed/disabled/student: 5.8 ± 1.4; P = .048), lower income (<US$25 000: 6.0 ± 1.2 to >US$75 000: 5.3 ± 1.3, P = .020), and fewer symptoms of depression (r: −0.20, P = .011; Table 1). Accounting for other variables using multivariable analysis, greater symptoms of depression was the only factor independently associated with lower satisfaction (β: −0.03; 95% CI: −0.05 to −0.00; P = .047, semipartial R ²: 0.047; adjusted R ² full model = 0.10; Table 2).

Table 1.

Bivariate Analyses of Factors Associated With GSS.^a

Variables	GSS	P value
Age (r)	0.16	.047
Sex
Women	5.8 ± 1.3	.006
Men	5.3 ± 1.3
Race/ethnicity
White	5.5 ± 1.3	.589
Non-white	5.6 ± 1.3
Marital status
Married/unmarried couple	5.5 ± 1.3
Single	5.4 ± 1.2	.073
Divorced/separated/widow	6.1 ± 1.2
Level of education
High school or less	6.3 ± 1.1	.002
Some college	5.6 ± 1.3
College graduate	5.5 ± 1.3
Masters’ degree or more	5.1 ± 1.1
Work status
Working	5.4 ± 1.3	.048
Retired	5.9 ± 1.3
Unemployed/unabled/student	5.8 ± 1.4
Income
<US$25 000	6.0 ± 1.2	.020
US$25 000-US$50 000	6.0 ± 1.3
US$50 000-US$75 000	5.4 ± 1.3
>US$75 000	5.3 ± 1.3
Insurance
Private insurance	5.4 ± 1.3	.066
Medicare	5.6 ± 1.3
Other or no insurance	6.1 ± 1.2
Sort visit
New patient	5.5 ± 1.2	.699
Follow-up patient	5.6 ± 1.4
Diagnosis
Trauma	5.7 ± 1.3	.179
Nontrauma	5.4 ± 1.3
PROMIS depression (r)	−0.20	.011
PCS-4 (r)	−0.04	.606
SHAI-5 (r)	−0.09	.242

Abbreviations: GSS, Guttman Satisfaction Scale; PCS, Pain Catastrophizing Scale; PROMIS, Patient-Reported Outcomes Measurement Information System; SHAI, Short Health Anxiety Inventory.

^a Boldface value indicates statistically significant difference; Spearman’s correlation indicated by r; continuous variables as mean ± standard deviation.

Table 2.

Multivariable Linear Regression Analyses of Factors Associated With Guttman Satisfaction Scale.^a

Dependent variables	Retained variables	Regression coefficient [β] (95% confidenc interval)	Standard error	P value	Semipartial R 2	Adjusted R 2
Guttman Satisfaction Scale	Age in years	0.00 (−0.02 to 0.02)	0.01	.948		0.10
	Woman compared to men	0.41 (0.04 to 0.86)	0.07	.071
	Marital status
	Married/unmarried couple	Reference value
	Single	0.04 (−0.53 to 0.62)	0.29	.883
	Divorced/separated/widow	0.28 (−0.33 to 0.89	0.31	.369
	Level of education
	High school or less	Reference value
	College graduate	−0.13 (−0.60 to 0.34)	0.24	.584
	Masters’ degree or more	−0.48 −1.06 to 0.10)	0.29	.104
	Work status
	Working	Reference value
	Retired	0.30 (−0.28 to 0.88)	0.29	.309
	Unemployed/unabled/student	0.06 (−0.67 to 0.78)	0.37	.881
	Income
	<US$25 000	Reference value
	>US$75 000	−0.15 (−0.61 to 0.32)	0.23	.53
	Insurance
	Private insurance	Reference value
	Other or no insurance	0.54 (−0.06 to 1.13)	0.30	.075
	PROMIS Depression	−0.03 (−0.05 to −0.00)	0.05	.047	0.02

Abbreviation: PROMIS, Patient-Reported Outcomes Measurement Information System.

^a Boldface value indicates statistically significant difference; only the semipartial R ² of significant variables is displayed.

Factors Associated With the Top 2 Possible Scores of GSS

Accounting for other variables using multivariable analysis, women (compared to men, odds ratio[OR]: 2.12, 99% CI: 1.01-4.45, P = .046) and high school or lower level of education (masters’ degree compared to high school or less; OR: 0.16, 95% CI: 004-0.61, P = .007) were independently associated with the top 2 possible scores of the GSS (insert stats), but symptoms of depression were not (Tables 3 and 4).

Table 3.

Bivariate Analyses of Factors Associated With the Top 2 Ratings of GSS.^a

Variables	Gutmann Satisfaction Score		P Value
	4-5	6-7
Age	49 ± 16	53 ± 17	.137
Sex
Women	33 (44)	48 (60)	.054
Men	42 (56)	32 (40)
Sort visit
New patient	35 (47)	43 (54)	1.00
Follow-up patient	40 (53)	37 (46)
Diagnoses
Trauma	40 (53)	39 (49)	.631
Nontrauma	35 (47)	41 (51)
Race/ethnicity
White	46 (61)	53 (66)	.616
Non-white	29 (39)	27 (34)
Marital status
Married/unmarried couple	48 (64)	45 (56)
Single	21 (28)	18 (23)	.061
Divorced/separated/widow	6 (8)	17 (21)
Level of education
High-school or less	6 (8)	24 (30)	.002
Some college	20 (27)	18 (23)
College graduate	28 (37)	27 (34)
Masters’ degree or more	21 (28)	11 (14)
Work status
Working	54 (72)	46 (58)	.156
Retired	14 (19)	25 (31)
Unemployed/unabled/student	7 (9)	9 (11)
Income
<US$25 000	7 (9)	15 (19)	.080
US$25 000-US$50 000	12 (16)	21 (26)
US$50 000-$75 000	15 (20)	13 (16)
>US$75 000	41 (55)	31 (39)
Insurance
Private insurance	54 (72)	46 (58)	.133
Medicare	14 (19)	19 (24)
Other or no insurance	7 (9)	15 (19)
PROMIS Depression	50 ± 8.6	47 ± 8.5	.039
PCS-4	4.1 ± 4.2	3.7 ± 4.0	.535
SHAI-5	9.7 ± 2.1	9.2 ± 2.9	.226

Abbreviations: GSS, Guttman Satisfaction Scale; PCS, Pain Catastrophizing Scale; PROMIS, Patient-Reported Outcomes Measurement Information System; SHAI, Short Health Anxiety Inventory.

^a Boldface value indicates statistically significant difference; mean ± standard deviation; discrete variables as number (percentage).

Table 4.

Multivariable Logistic Regression Analyses of Factors Associated With the Top 2 Ratings of GSS.^a

Dependent variables	Retained variables	Odds ratio (95% confidence interval)	Standard error	P value	C statisticb
Top Guttman Satisfaction Scale Ratings	Woman compared to men	2.12 (1.01 to 4.45)	0.80	.046	0.74
	Marital status
	Married/unmarried couple	Reference value
	Single	0.98 (0.40 to 2.4)	0.45	.964
	Divorced/separated/Widow	2.0 (0.65 to 6.2)	1.2	.228
	Level of education
	High school or less	Reference value
	Some college	0.20 (0.06 to 0.66)	0.12	.008
	College graduate	0.31 (0.10 to 0.97)	0.18	.045
	Masters’ degree or more	0.16 (0.04 to 0.61)	0.11	.007
	Income
	<US$25 000	Reference value
	US$25 000-US$50 000	0.86 (0.25 to 2.9)	0.54	.808
	US$50 000-US$75 000	0.49 (0.13 to 1.8)	0.32	.280
	>US$75 000	0.64 (0.19 to 2.2)	0.40	.475
	PROMIS Depression	0.98 (0.94 to 1.0)	0.02	.287

Abbreviations: PROMIS, Patient-Reported Outcomes Measurement Information System; SHAI, Short Health Anxiety Inventory.

^a Boldface value indicates statistically significant difference.

^b The C statistic is a measure of model fit and is the area under the receiver operating characteristics curve.

Floor Effect, Ceiling Effect, Skewness, and Kurtosis of the GSS

The GSS had no floor effect and a ceiling effect of 38% (Supplementary Appendix 1). The lowest score was 4, which means that none of the patients disagreed with the first statement: “Today’s visit was satisfactory.” Patients with a score of 4 disagreed with the second statement: “Today’s visit was more satisfactory than usual.” Thirty-eight percent of the patients agreed with the most satisfactory statement: “Today’s visit was more satisfying than I could have imagined.” The skewness was −0.08, consistent with a disproportionate number of highest possible values (a long left tail). The kurtosis was 1.3, where a normal distribution is 3—a relatively non-normal distribution.

Discussion

Patient satisfaction is often considered a measure of health care quality (29). An understanding of what makes a clinical encounter satisfying might improve adherence, improve health, and decrease complaints. The large ceiling effect of satisfaction questionnaires and other experience measures hinders learning and growth via experiments (clinical research and quality improvement). It would be ideal to measure satisfaction independent of other factors without censoring and to have opportunity to learn from various levels of satisfaction (10,13). We studied factors independently associated with the Guttmann style satisfaction questionnaire and the top 2 possible scores of this questionnaire. Also, we measured the floor effect, ceiling effect, the skewness, and kurtosis of the GSS.

Our results should be interpreted in light of some limitations. First, these data may apply best to an orthopedic specialty visit. The data are collected in 1 city in multiple different offices. Also, we excluded non-English speakers because we did not have all questionnaires validated in Spanish or other languages. We included both new and return patients. Return patients may be taking into consideration other aspects of care when completing the satisfaction survey. However, according to our results, there were no differences in satisfaction between new and return patients in bivariate analysis.

The small correlation between symptoms of depression and satisfaction is consistent with prior evidence (4,30,31). Patients after open carpal tunnel release had a relatively small correlation between depression and satisfaction (r = −0.24) (30). Cognitive coping strategies and heightened illness concerns have no correlation with satisfaction in this and in prior studies (32). Given that the prevalence of depression is higher in people seeking care than in the general public (31,33,34), clinicians ought to prepare and plan for a corresponding experience of dissatisfaction with care.

The factors associated with the top 2 possible scores of the GSS, high school or less education and female gender, are not consistent with prior research (4,10,14,30,35). For example, among 108 patients visiting an orthopedic hand surgeon, gender was not independently associated with the top 2 possible scores of patient satisfaction with the surgeon (4). In another study of 178 orthopedic patients, education was not significantly associated with patient satisfaction with the visit (35). The associations with demographic factors are small and may be spurious pending additional study.

The 38% ceiling effect of the GSS is lower than ordinal satisfaction scales but still undesirably high. A ceiling effect occurs when the responses are not evenly distributed and show a positive “skew” toward the favorable end (36). When a ceiling effect occurs, there could still be a normal distribution, with information lost at the top of the scale due to the measure’s inability to stratify very high ratings. There are satisfaction questionnaires with a lower ceiling effect, for example, the Press Ganey Medical Practice Survey (37) with a ceiling effect of 29.3%. This questionnaire has 24 items organized into 6 scales: access, moving through the visit, nurse assistant, care provider, personal issues, and overall assessment. However, patients with the GSS have to answer a maximum of 4 questions. It would be ideal to measure satisfaction independent from other factors, using a few questions or, even better but probably impossible, a single question with no demonstrated ceiling effect for scores.

Skewness and kurtosis are rough indicators for a normal distribution of values (38,39). Skewness is an index of the symmetry of a distribution (38,39). The skewness was −0.08, which indicates many very high values (a long left tail). Kurtosis is a measure to describe tailedness of a distribution (38,39). The kurtosis of a normal distribution is 3 and we found a kurtosis of 1.3, which means this GSS has a relatively non-normal distribution.

It is difficult to think of other measures for which researchers would accept such high ceiling effects. Because the Guttman scale created some spread in the data, we were able to assess satisfaction on its continuum and measure correlations rather than categorizing satisfaction. This could have advantages. For instance, symptoms of depression correlated on the continuum but were not associated when satisfaction was treated as a dichotomous variable. Since both the experience of satisfaction and symptoms of depression occur on a continuum, information may be lost if we categorize or dichotomize these factors. A sequential series of statements or questions (Guttman style) shows potential to limit ceiling effects, but additional strategies are needed (40). The questions can likely be refined with more options at the top end and more superlative final options. Another strategy might be to remind people of their agency and help them feel comfortable that their honest appraisals are being solicited in order to help clinicians improve. Finally, it might turn out that verbatim descriptions of how clinicians can improve specific aspects of care might prove more useful than patient-reported satisfaction measures. These ideas represent areas for future research.

Author Biographies

Yvonne Versluijs is a Dutch medical doctor and was a visiting scholar at the Department of Surgery and Perioperative Care, Dell Medical School—University of Texas, Austin at the time of this research.

Laura E Brown has a PhD in Communication and focuses on interpersonal communication, health communication in particularly. She was the co-director of the Center for Health Communication Think Tank at the Dell Medical School and Moody College of Communication—University of Texas, Austin and she is now the director of UTNY.

Mauna Rao was a medical student researcher at the Department of Surgery and Perioperative Care, Dell Medical School—University of Texas, Austin at the time of this research.

Amanda I Gonzalez is a Swiss hand surgeon and was a visiting scholar at the Department of Surgery and Perioperative Care, Dell Medical School—University of Texas, Austin at the time of this research.

Matthew D Driscoll is an orthopedic surgeon and sports medicine specialist at Austin Regional Clinic and an assistant professor of Orthopedic Surgery at Dell Medical School—The University of Texas, Austin, Texas.

David Ring is an orthopedic surgeon, associate dean for comprehensive care, and professor of surgery and psychiatry at Dell Medical School—The University of Texas, Austin, Texas.