Understanding the Determinants of Patient Satisfaction with Surgical Care Using the CAHPS Surgical Care Survey (S-CAHPS)

Abstract

Background

Patient satisfaction has been increasingly emphasized, including the use of financial penalties for underperformance. However current measures of patient satisfaction do not address aspects specific to the care of surgical patients. We therefore aimed to examine the recently validated Surgical Care Survey (S-CAHPS) in order to determine which aspects of perioperative care are predictive of satisfaction with the surgeon.

Methods

All patients undergoing a general surgery operation at our institution over a 5 month period were sent a modified S-CAHPS within 3 days of discharge. Patients were then divided into two groups: those that rated their surgeon as the best possible and those giving a lower rating. Univariate and multivariate analyses were used to determine predictors of satisfaction with surgical care. S-CAHPS results were then compared with other satisfaction measures in a subset of patients.

Results

The response rate was 45.3% (456/1007). The average age was 59±16 years, length of stay was 4.1±6.6 days, and 23% had unscheduled operations. 72% of patients rated their surgeon as the best surgeon possible. On multivariate analysis, preoperative communication and attentiveness on the day of surgery were the most important determinants of overall surgeon rating. S-CAHPS scores correlated with other standard measures of satisfaction (HCAHPS scores).

Conclusions

S-CAHPS is a novel surgical satisfaction tool and is feasible to administer to patients undergoing general surgical procedures. Surgeon characteristics most predictive of high patient satisfaction are effective preoperative communication and attentiveness on the day of surgery.

Introduction

As one component of the Affordable Care Act (ACA), the National Quality Strategy (NQS) brought patient-centered health care to the forefront by making it central to the first of its three aims.¹ Efforts to emphasize patient-centeredness required the use of validated patient experience-of-care survey metrics to better measure these attributes of clinical care. The Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys, originally designed in 1995 to examine healthcare plans, were modified in 2002 to allow public reporting of patient satisfaction at the hospital level. The Centers for Medicare and Medicaid (CMS) partnered with the Agency for Healthcare Research and Quality (AHRQ) to design and test what is now known as the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey.²

The medical community’s interest in patient satisfaction data was clearly stimulated, at least in part, by the initiation of public reporting of HCAHPS data in 2008. Although useful to patients, the public reporting of hospital level patient satisfaction data did not provide guidance at the level of individual clinicians or practice groups. However, the ACA also included future mandates that required the comparison of individual physicians using patient experience measures.^3,4 In order to implement this effort, the Clinician and Group CAHPS (CG-CAHPS) survey was chosen as the appropriate metric for such system wide comparisons.

Early in this process, the American College of Surgeons (ACS) formed the Surgical Quality Alliance in partnership with other surgical organizations and examined the potential application of CG-CAHPS for measurement of patients’ perceptions of surgeons specifically. Several previous studies had demonstrated the generally higher satisfaction scores in surgical compared to medical patients.^5,6 This finding was so consistent that it was incorporated into standard case mix adjustment for HCAHPS scores.⁷ Recognizing the fundamental differences in the delivery of surgical and medical care, concerns were raised that the metrics used to measure the patient experience for surgical patients might not be adequately captured by more global instruments. In that context, several groups began to study and develop surgery-specific measures to examine patient satisfaction.^8–11

In keeping with previous findings, the Surgical Quality Alliance concluded that the CGCAHPS survey did not adequately capture information most relevant to surgical patients. As a consequence, they then embarked upon the development of a CAHPS survey that would address elements of the surgical experience most relevant to patients. After extensive design and evaluation, the resulting survey – the CAHPS Surgical Care Survey or S-CAHPS – was endorsed by the National Quality Forum and released for use in June of 2012.¹² The survey development and validation process mirrored that of the parent CAHPS surveys and included stakeholder engagement, comprehensive literature review, critical incident analysis, focus groups, cognitive interviewing and extensive field testing. Because the survey was not designed to be specialty specific, the field testing only included 397 patients (14% of all survey respondents) from 11 individual general surgeons. Since that time, there has been only one published report describing use of the S-CAHPS in a clinical setting (i.e. amongst otolargyngology patients).¹³

Therefore, the aims of this study were first to confirm the feasibility of survey administration to a large population of general surgical patients in a real-world clinical setting and second, we aimed to determine which specific domains of surgical care correlated most strongly with overall surgeon satisfaction, in order to identify areas of potential interest for targeted quality improvement efforts.

Methods

Patients

All discharged adult (≥18 years), English speaking, non-prisoner patients who had an operation at the University of Wisconsin Hospital and Clinics with a general surgeon between June and November of 2013 were mailed a survey within 3 days of discharge. No incentives were given for survey completion, but a cover letter detailing the purpose of the survey was included. These patients were identified by sorting the daily hospital discharge list by general surgery service and surgeon. Patients who were admitted by a surgeon but did not undergo an operative procedure during that admission were not mailed surveys. Patients who did not have an overnight admission were also not surveyed. Those that incorrectly identified the surgeon on the returned survey were excluded. Secondary mailings with reminders were sent to nonrespondents for the first 226 patients. However secondary response rates remained low and secondary mailings were therefore discontinued. Survey responses were tabulated and entered into an ongoing database, as part of a prospective study examining surgical care, readmissions, and satisfaction. Retrospective chart reviews were undertaken to gather a variety of additional clinical variables. This study was undertaken after institutional review board approval.

Survey

The S-CAHPS survey published by the Agency for Healthcare Research and Quality (AHRQ) was modified to focus on preoperative and day of surgery care, and the questions about postoperative issues were therefore omitted (questions 26 – 34 in the complete survey). Given that we were also interested in studying the relationship of postoperative satisfaction and readmission, we chose to capture satisfaction with preoperative and inpatient care just after discharge, before postoperative readmissions were likely to have occurred. We examined the composite domains outlined by the S-CAHPS instructions for use: information to help you prepare for surgery (Preparation for Surgery), how well surgeon communicates with patient before surgery (Preoperative Communication), and surgeon’s attentiveness on day of surgery (Day of Surgery Attentiveness). Individual questions comprising each of these composite domains are given in Table 1. Because no composite variable examining shared decision-making was validated in the design of the SCAHPS survey, we selected one question (discussion about reasons not to have surgery) to include in the analysis based on its clinical importance. Further, as patients undergoing urgent surgery comprise a significant fraction of surgical cases, we opted to include them in our sample. However, because this group of patients did not have a clinic visit before surgery, they were not included in the analysis of preoperative preparation and communication, and are also excluded from the multivariate analysis presented in Table 4.

Table 1.

Comparison of Respondents and Nonrespondents

Variable	Respondents (n=456)	Nonrespondents (n=551)	P value
Age (mean±SD)	59±16	50±17	<0.0001
Female (%)	61%	57%	NSD
Length of Stay (days±SD)	4.1±6.6	6.0±8.8	0.0002
Caucasian (%)	91%	90%	NSD
Surgeon Group (%)
Burn/Acute Care/Trauma	27.4%	43.7%	<0.001
Endocrine	11.1%	7.3%	0.031
Colorectal	26.5%	20.5%	0.024
Hepatobiliary	7.9%	6.9%	NSD
Breast	10.1%	6.0%	0.016
Minimally Invasive	16.9%	15.6%	NSD

Note: The abbreviation of “SD” denotes standard deviation and “NSD” indicates that the difference was not significant.

Table 4.

Multivariate Analysis for Top box Rating for Surgeon

Variable	Odds Ratio [95% CI]	Wald χ2	P value
Age	1.022 [1.00,1.04]	4.3202	0.0438
Surgeon Group	-	4.6221	0.4637
Preparation for Surgery	2.543 [1.1, 6.1]	4.3202	0.0377
Preoperative Communication	6.826 [2.9, 16.1]	19.3088	<0.0001
Day of Surgery Attentiveness	5.387 [2.2, 13.4]	13.1	0.0003
Shared Decision Making	-	8.9524	0.0299

Note: The multivariate analysis excludes the 100 patients undergoing urgent surgery. Another 46 patients were excluded as they had some missing values for the questions necessary for calculation of the composite scores. As such, a total of 289 patients are available for this multivariate analysis.

As we were investigating patient-level data, we calculated composite scores as follows. Patients who indicated the best possible response for each of the individual questions from which the composite is comprised were designated as top box respondents for that composite measure (e.g. for day of surgery attentiveness domain to be considered top box, the respondent had to have answered the top box response for both questions). All other variations of responses were grouped in the not top box category. Further, if a response to any of the composite questions was not given, the composite domain was unable to be calculated, and that particular patient was not included for that domain. Patients were then divided into groups for analysis based on how they responded to the global assessment of the surgeon. Specifically, the question asked was “Using any number from 0 to 10, where 0 is the worst surgeon possible and 10 is the best surgeon possible, what number would you use to rate all your care from this surgeon?” For this question, a response of 10 was considered the top box response, and patients were again grouped into those who gave the top box response and those who gave anything else (scores of 0–9). Although standard HCAHPS top box scores for hospital evaluation include scores of 9 and 10 on this scale, we chose only to look at 10 on the S-CAHPS survey question to improve the discriminatory ability of the comparisons given the significant right skew in the data. Sensitivity analyses were conducted using 9 or 10 as a top box outcome and showed similar findings to those presented here.

The S-CAHPS was also compared with other patient satisfaction measures including the HCAHPS, Press Ganey, (an HCAHPS precursor) and the Ambulatory Patient Experience Survey Results (Avatar) surveys. For HCAHPS and Press Ganey patients who also completed the S-CAHPS, either of these surveys was analyzed using correlations. For the Avatar survey, our hospital does not release individual patient data; therefore, the average surgeon score during the study period was used as a surrogate metric. This was then was compared with the average S-CAHPS overall surgeon rating using correlations, with the unit of analysis being the surgeon.

Statistical Analysis

Data analysis was generated using SAS 9.3 software (SAS Institute, Cary NC), with appropriate application of χ² and t-tests for univariate analysis. Multivariable logistic regression was also generated using SAS. Urgent operation and surgeon group were highly correlated, as would be expected based on practice patterns of the various specialties. Given this significant correlation, we elected to include surgeon group and exclude urgent operation from the multivariable logistic regression model.

Results

Feasibility

The response rate for the 1007 surveys sent was 45.3% (456/1007). Twenty-one of these patients were excluded because they did not have an overnight stay, incorrectly identified the operating surgeon or did not undergo a general surgery operation. The final sample size was thus 435 patients. They were operated upon by one of 28 surgeons in the Division of General Surgery. Additionally, 27.1% of patients were treated by colorectal surgeons, 26.7% by burn/trauma/acute care surgeons, 16.7% by minimally invasive surgeons, and the remainder by breast, hepatobiliary, and endocrine surgeons.

Characteristics of Survey Respondents

The average age of the respondents was 59±16 years and 61% were female. 37.5% of patients were admitted only overnight with the remainder having longer inpatient stays. The average length of stay was 4.1±6.6 days. Nearly 23% (100/435) of patients had unscheduled operations and had no preoperative outpatient visit with the surgeon and were therefore excluded from calculation of preparation for surgery, shared decision making, and preoperative communication domains. Only 10% of patients had no prior surgery, with 21% of patients having more than six previous operations. Self-reported health was excellent or very good in 48% of patients and fair or very poor in only 16%. The vast majority of patients (93%) were able to complete the survey without assistance; of those that needed assistance, most (60%) only had someone read the questions aloud to them.

Respondents and nonrespondents were similar in terms of gender (61% vs. 57% female, respectively), and race (91% vs. 90% Caucasian, respectively). Nonrespondents had a significantly longer length of stay (6.0 vs. 4.1, p=0.0002) and were younger (50 vs. 59, p<0.0001) than those who responded (Table 1). Among nonrespondents, there was an overrepresentation of patients undergoing surgical procedures by the emergency general surgeons or surgery for trauma. Said another way, the response rates were significantly lower in patients undergoing emergency surgery or surgery for trauma when compared to patients undergoing elective surgery by subspecialists (34% vs. 52%, p <0.0001)

Survey Results

As seen in other versions of CAHPS surveys, there is a significant rightward skew to patient responses.^14,15 For both questions relating to preparation for surgery, at least 89% of respondents gave top box scores. Similarly for all four questions related to preoperative communication, more than 90% of patients gave top box responses. For questions related to attentiveness on the day of surgery, there was more variation. Only 85% of patients said that the surgeon discussed the outcome of surgery with them before leaving the hospital, while 93% said the surgeon visited with them just before the operation (Table 2).

Table 2.

S-CAHPS Composite Domains Top box Response Percentages

Preparation For Surgery Domain	Possible Responses	Top box %
Before your surgery, did anyone in this surgeon’s office give you all the information you needed about your surgery?	Yes, definitely* Yes, somewhat No	89%
Before your surgery, did anyone in this surgeon’s office give you easy to understand instructions about getting ready for your surgery?		90%
Shared Decision Making Domain
During your office visits before your surgery, did this surgeon talk to you about the reasons you might not want to have the surgery?	A lot* Some A little Not at all	13%
Preoperative Communication Domain
During your office visits before your surgery, did this surgeon listen carefully to you?	Yes, definitely* Yes, somewhat No	94%
During your office visits before your surgery, did this surgeon spend enough time with you?		91%
During your office visits before your surgery, did this surgeon encourage you to ask questions?		91%
During your office visits before your surgery, did this surgeon show respect for what you had to say?		96%
Day of Surgery Attentiveness Domain
After you arrived at the hospital or surgical facility, did this surgeon visit you before your surgery?	Yes* No	93%
Before you left the hospital or surgical facility, did this surgeon discuss the outcome of your surgery with you?		85%

Top box response indicated by *

In contrast, the fraction of top box respondents was much lower for questions relating to shared decision making. When asked if surgeons discussed more than one way to treat their condition, only 69% said yes. Nearly 80% of patients said surgeons asked them which treatment option they thought would be best. Most strikingly, while 72% of patients said their surgeon talked to them about reasons they might want to have surgery, only 13% of patients said their surgeon talked about reasons they might not want to have surgery.

On the overall surgeon-rating item, 72.4% rated their surgeon as the “best surgeon possible” (10 on a scale of 0 to 10). In contrast, only 58.7% of patients gave the highest rating to their anesthesiologist.

Determinants of Satisfaction with Surgeon

On univariate analysis (Table 3), older patient age and elective surgery were significantly associated with higher surgeon ratings. Of the 315 patients that rated the surgeon as the best possible, approximately 245 were included in each of the preoperative composites (communication, preparation for surgery, shared decision making) with exclusions of patients undergoing urgent surgery or failure to answer questions required to calculate composite scores. In all, 100 patients (from the total cohort of 435 patients) had an urgent surgery, and were therefore excluded from the preoperative composite domain calculations (Table 3). Not surprisingly, each of the pre-specified composite domains (preparation for surgery, preoperative communication, and day of surgery attentiveness) was significantly correlated with higher overall surgeon ratings (p<0.0001 for all). Patient participation in shared decision making as measured by the selected question (discussion about reasons to not have surgery) was also associated with higher surgeon rating, but to a much lesser extent (p=0.0252). Importantly, neither self-reported health status nor the number of previous operations was correlated with overall surgeon satisfaction. Also, length of stay (top box 4.1±6.1 days vs. not top box 4.0±8.1; p = 0.7906) and education level (% with some college: top box 66.4% vs. not top box 66.9%; p = 0.4561) were not different between the groups.

Table 3.

Univariate Analyses of Top box Rating for Surgeon

Variable	Patient with Highest Overall Surgeon Rating (n=315)	Patients with Lower Overall Surgeon Rating (n=120)	P value
Age (mean±SD)	60±16	56±16	0.014
Surgeon Group	-	-	<0.0001
Urgent Operation (%)	16.2% (n=315)	40.8% (n=120)	<0.0001
Preparation for Surgery (% top box response)	91.1% (n=248)	65.1% (n=63)	<0.0001
Preoperative Communication (% top box response)	92.7% (n=247)	62.3% (n=61)	<0.0001
Day of Surgery Attentiveness (% top box response)	88.0% (n=309)	57.3% (n=117)	<0.0001
Shared Decision Making (% top box response)	18.7% (n=235)	4.8% (n=62)	0.0252

Note: The abbreviation of “SD” denotes standard deviation. The number (n=) of patients for each group represents the total number of patients in each group that was included in the calculation of the percentage. Please see text for details.

A multivariate analysis (Table 4) was then undertaken to examine the relative strength of the associations found above. Preoperative communication was very strongly associated with higher surgeon rating (OR 6.8 [2.9–16.1], p < 0.0001). This was also true for day of surgery attentiveness (OR 5.4 [2.2–13.4], p = 0.0003). Shared decision making and preparation for surgery were less strongly associated. Age remained significant (p = 0.0483), but surgeon group was not an important predictor of satisfaction with the surgeon in the multivariate model. Because the multivariate analysis excluded patients having urgent surgery, this model was also run including those patients to ensure that this did not affect the results. With their inclusion in the model, there were no differences in the variables correlated with surgeon rating, and the p values for each domain were nearly identical (data not shown).

In order to help evaluate the utility of top box scores in each of the individual composite domains, positive predictive values (PPV), negative predictive values (NPV), and accuracy for each were calculated (Table 5). As can be seen in the table, the PPV’s were much higher than the NPV’s for each domain. Overall, accuracy rates were on the order of 80% for each domain. While preoperative communication had the highest values for PPV, NPV, and accuracy, the differences between the values for each domain were small.

Table 5.

Assessment of Positive Predictive Value, Negative Predictive Value and Accuracy

Variable	Positive Predictive Value	Negative Predictive Value	Accuracy
Preparation for Surgery	84.6%	50.0%	79.7%
Preoperative Communication	85.7%	56.1%	81.8%
Day of Surgery Attentiveness	83.3%	53.6%	79.6%

Comparison with Other Patient Satisfaction Measures

A fraction of the patients, included in this study, also completed additional patient satisfaction surveys. When evaluating for correlation between the S-CAHPS overall surgeon rating and physician communication measures for the HCAHPS and another hospital-level survey, Press Ganey, we found a small but significant correlation between the respective physician rating domains (HCAHPS Physician Communication Composite: R = 0.321, p = 0.003, n = 87; Press Ganey physician rating: R = 0.369, p = 0.001, n =77) (Figure 1). However, there was no significant correlation using two physician rating elements from the outpatient Avatar survey, although this conclusion is limited based on the sample size (physician composite: R = 0.301, p = 0.1967, n = 20; recommend physician: R = 0.301, p = 0.197, n = 20).

Figure 1. Correlation of S-CAHPS with other Satisfaction Measures.

Other survey ratings of physician communication and overall physician rating

A. S-CAHPS demonstrates a statistically significant correlation with two inpatient satisfaction measures – the HCAHPS physician communication composite (n=87) and the Press Ganey overall physician rating (n=77). B. Individual outpatient patient satisfaction measures were unavailable, therefore the average Avatar rating for both likelihood of recommending the physician and communication composite for a specific surgeon during the studied period was compared with the average S-CAHPS for that surgeon. There were 20 surgeons that had both S-CAHPS and Avatar scores, and there was not a significant correlation between these measures.

Discussion

Central Findings

This study has demonstrated the feasibility of administering the S-CAHPS survey in the clinical setting for patients undergoing a variety of general surgical procedures. In addition, we have shown that effective preoperative physician-patient communication is an important predictor of overall satisfaction with the surgeon. To our knowledge this is only the second published report examining S-CAHPS, and is the largest report in the literature examining this new metric designed specifically to measure the surgical patient experience.¹³ These data emphasize specific actionable domains that surgeons can target in efforts at ongoing quality improvement.

Comparison of Present Data with Others

The overall response rate for the S-CAHPS survey in our study was 45.3%. This is higher than our institutional response rate for the HCAHPS survey of 34% over an equivalent time period. Further, for survey sites with 20 or more providers (like ours), a minimum of 300 completed surveys are recommended for analysis. Our study includes 435 patients with completed surveys. Our overall response rate is nearly identical to that described by Schulz et al.¹³ It is however somewhat lower than the overall response rate of 49% obtained during the piloting of the survey by the Surgical Quality Alliance.¹⁶ It is however similar to that for the general surgery cohort (42.5%) in that pilot.¹⁶ Further, our response rate should be viewed and compared in light of the fact that we included patient admitted for emergent surgery. In this group, the response rate of 34% was much lower than the rate in the remaining patients undergoing scheduled procedures (52%). Nevertheless, the response rate in this study is in excess of the 40% response rate recommended by the CAHPS consortium, as well as the 32% response rate for HCAHPS data nationally¹⁷.

Although our response rate is well within the range accepted for this survey type, the effect of nonrespondents on the outcome should be considered. It has previously been established that failure to complete the HCAHPS survey is more common among males, non-Hispanic whites, and those with lower self-reported health status.^18,19 It has also been demonstrated indirectly that nonrespondents tend to have somewhat lower satisfaction with healthcare^18,20, which raises concern about the need to adjust absolute survey scores downward to truly reflect the whole population of patients at a given facility. This issue has been thoroughly examined as part of the CAHPS II project.¹⁸ These results demonstrated that the amount of score inflation as a result of nonrespondents being less satisfied was small and two thirds of that was eliminated simply by standard case mix adjustment. Further, adding nonresponse weighting would only improve estimates of satisfaction (i.e. the mean squared error) only when more than 1000 completed surveys are examined. For these reasons, we would maintain that our results are a very close approximation of satisfaction amongst the entire patient population surveyed.

With regard to overall surgeon satisfaction, 72% of our patients gave their surgeon the highest rating available (10 on a scale of 0 to 10). This is nearly identical to that found in the survey pilot data (71%), and that found in patients undergoing head and neck surgery (approximately 74%).^13,16 While top box scores were given more often by the general surgery patients (when compared to the otolargngology patients) across all comparable domains, both groups of patients gave the lowest ratings to surgeon attentiveness the day of surgery. Finally, top box scores for anesthesiologists are similar for our study (59%) and the survey pilot data (54%).¹⁶

We also examined whether the results of the S-CAHPS was correlated with other patient satisfaction measures that are currently in wide-spread use. We found a modest, but significant correlation between inpatient physician composite measures using the HCAHPS and Press Ganey Surveys. However, these results were not reflected on the outpatient Avatar physician measures. This finding suggests that there is an inconsistent correlation between the S-CAHPS and other measures, but this conclusion is limited by a small number of patients that responded to both the S-CAHPS and one of the other patient satisfaction measures, as well as the methodology used for Avatar comparisons. Further exploration in this area is needed.

An interesting finding in this study was the relative lack of shared decision making in surgical consultations as measured by four separate items. These items all had low top box percentages compared to all other surgeon related questions. In fact, the S-CAHPS survey has not yet worked out a reliable composite for this domain because of the inconsistency of survey respondents.¹⁶ However, because of the theoretical importance of shared decision making in the process of surgical care, S-CAHPS has retained these questions and encourages further study in this area. Our results were surprising in that only 13% of patients reported discussing reasons not to have an operation with their surgeon, whereas 72% discussed reasons to consider surgery. These data raise concerns about the practical execution of shared decision making in surgical evaluation, and this area is also ripe for further study.²¹

Determinants of Satisfaction with Surgeon

The central finding in this study is the strong predictive value of the preoperative communication domain with overall rating of the surgeon. The effect of physician-patient communication on patient satisfaction has been previously studied.^22–24 Although the data are heterogeneous and primarily focused on medical patients, the bulk of the evidence supports a strong link between effective physician communication and high patient satisfaction. The positive influence of effective physician-patient communication preoperatively on patient satisfaction has been nicely demonstrated in a recent surgical population as well.²⁵

The mechanisms by which effective communication begets better patient outcomes and improved satisfaction are not clear.²⁶ Our finding that the preoperative communication domain is more closely associated with high satisfaction than those aspects of communication that relate to preparation for surgery or shared decision making suggests that it is by way of the interpersonal skills of the surgeon.²⁷ In other words, it is the ability of the surgeon to connect with patients (i.e. to listen carefully, encourage questions, show respect, and spend time with patients) that is highly valued by patients. Given that patients must enter into explicitly trusting relationships with their surgeons in order to agree to undergo any surgical procedure, it is intuitive that interpersonal skills are especially critical for surgeons.

Limitations

The primary limitation of the present report is the inclusion of only preoperative and hospital survey questions, rather than the complete S-CAHPS survey. Although this information would be useful on one level, the postoperative communication domains were so highly correlated with those same domains preoperatively in the pilot data16 that this is unlikely to have changed our conclusions. Further, surveying patients after a more delayed period to capture postoperative events raises concerns about the potentially negative effect those events (e.g. readmission or complications) may have on satisfaction with the preoperative and day of surgery care. Thus, surveying patients immediately after discharge serves to minimize recall biases. Secondly, we included in our population a group for whom the survey was not intended - patients undergoing urgent surgery. These patients were included as they represent a significant fraction of patients in an academic medical center and nearly a quarter of the population in our sample. Because this subgroup is more difficult to study and presents multiple barriers for general surgeons, we elected to include them in our sample to examine use of this survey in a real-world clinical practice. However, given the relatively small numbers of such patients studied herein, the data remain exploratory in this area, and further study is needed.

As with any survey related research, another potential limitation relates to the survey’s ability to produce a valid quantitative representation of the intended qualitative construct, in this case “satisfaction.” Although the S-CAHPS was carefully developed and extensively tested with attention to all relevant psychometric issues, the ultimate demonstration of construct validity for patient “satisfaction” is challenging. Because patient satisfaction is in part reflective of individual patient expectations and has a variety of connotations, it is difficult to be certain that the survey itself effectively operationalizes a concept as elusive as patient satisfaction. Although the S-CAHPS does not reflect the full scope of the patient’s experience with an episode of surgical care, it is nonetheless a valid measurement tool for several important and standard aspects of the surgical patient experience. Further, there is no accepted gold standard to which this survey can be compared. Insomuch as qualitative interviewing represents the best methodology for determining satisfaction, the cognitive interviews conducted as part of the validation process demonstrate the fidelity of the survey to the construct of satisfaction.

Another important limitation of this study is the fact that it was conducted at a single center with a relatively homogenous group of patients from a similar demographic and socioeconomic background. We therefore are not able to generalize our findings to centers with a more diverse patient population. However, if comparisons were being made between centers (such as is commonly done for HCAHPS), case mix adjustment would largely mitigate their impact on the results. Because this was a single center study and the case mix was similar between comparison groups, we elected not to adjust the results for case mix here. Further, given that specific adjustment factors are not yet available specifically for the S-CAHPS, more response data across multiple centers is needed before accurate case mix adjustment can be accomplished.

Although we analyzed survey responses from more than 400 general surgical patients, our sample size is still relatively small relative to the potential pool of respondents on a national level. This undoubtedly limits our conclusions, as important differences may be seen when a broader and larger patient population is examined. Despite this, our analysis represents an important step in the understanding of the metrics of the SCAHPS and represents the first published report of its use in general surgical patients. Further, our data certainly bolster the claim of external validity made by the developers of the S-CAHPS, and in that way can be viewed as a demonstration of survey effectiveness.

Finally, as with many patient satisfaction surveys there was a significant rightward skew of the data, with the majority of patients giving relatively high scores for satisfaction. As with the HCAHPS survey, this tendency is not unique to our population. The effect of the S-CAHPS survey’s response type and scale on rightward skew was specifically studied and analyzed as part of the pilot’s field testing. Further, because we examined only dichotomized outcomes (i.e. top box vs not-top box), statistical testing is still valid despite the rightward skew in the data. Finally, we designated our top box group to be only those giving scores of 10, rather than 9 and 10, to help distinguish the group with highest satisfaction from all others.

Future Directions

Although the development and availability of S-CAHPS represents significant progress in the measurement of patient satisfaction for surgical patients, there remains a lot to be learned in this arena. Specifically, the effects of a variety of patient (e.g. benign vs malignant disease), surgeon (e.g. surgical volume) and system (e.g. multidisciplinary teams) factors on satisfaction amongst surgical patients are yet to be fully understood.²⁸ In addition, recent studies that examine the relationship of surgical quality metrics to patient satisfaction have used pooled (medical and surgical) HCAHPS data as an outcome variable, a practice which has important limitations.^29,30 Given this information, we plan to examine S-CAHPS results in response to more traditional outcomes measures, such as NSQIP-defined complication rates. The broad application of S-CAHPS to clinical practice if successful will allow much more meaningful outcome data for these and other studies.

Conclusion

In conclusion, S-CAHPS is an instrument that is feasible to administer in clinical practice and captures unique aspects of surgical care. Communication between patients and surgeons remains essential and should be increasingly emphasized in training and continuing education for surgeons. Use of the S-CAHPS in studies examining satisfaction among surgical populations should be encouraged.