Abstract
Background:
The increased focus on patient satisfaction has led to growth in the use of physician rating websites.
Purpose:
To analyze the factors associated with online 5-star patient reviews for orthopaedic sports medicine surgeons.
Study Design:
Cross-sectional study.
Methods:
A total of 70 orthopaedic sports medicine surgeons were randomly selected from the AOSSM website. A search was performed for these surgeons on Yelp.com. All reviews other than 5 stars (of a possible 5 stars) were excluded from the study. Each review was categorized as referring to a surgical or nonsurgical aspect of care, and each comment within the review was categorized as being clinically or nonclinically related. Comments were further subcategorized by specific features such as bedside manner, clinical outcomes, and patient education. Categorical variables were analyzed using the chi-square test.
Results:
Overall, 400 five-star reviews were included in the study, comprising 1225 total positive comments. Of the 400 five-star reviews, 200 (50%) were from surgically treated patients, and 200 (50%) were from nonsurgically treated patients. Of the 1225 positive comments, 505 (41%) were clinically related, and 720 (59%) were nonclinical. The most common positive clinical comments were for clear treatment plans (191 reviews [48%]), good outcomes (173 reviews [43%]), and providing alternative treatment plans (55 reviews [14%]). The most common positive nonclinical comments were for good physician bedside manner (287 reviews [72%]), friendly/professional staff (194 reviews [49%]), and ease of scheduling (68 reviews [17%]).
Conclusion:
The majority of 5-star patient reviews left positive comments regarding nonclinical aspects of care such as physician bedside manner and friendly staff. The most common positive comments regarding clinical aspects concerned good outcomes and clear treatment plans. The overall most common positive comment, in both surgically and nonsurgically treated patients, referred to good bedside manner.
Keywords: physician reviews, sports medicine, orthopaedic surgery, positive reviews, patient satisfaction
In recent years, there has been an increased focus on patient satisfaction. In 2010, the Affordable Care Act allowed the public to access data on patient satisfaction with hospitals and providers, which offered valuable information to help patients make educated choices about their health care.2,17,27 While government-funded websites such as HealthData.gov and the US Department of Health and Human Services Protect Public Data Hub are publicly available, only 14% of patients know about these resources.12,24 Patients more commonly rely on public physician review websites (PRWs), such as Yelp, Healthgrades, Vitals, and others, to make choices regarding their health care. 9 Specifically, a recent study reported that 59% of patients regard physician ratings as being important in their decision to choose a provider. 9 Furthermore, 54% of young adults rely on PRWs before going to see a physician, and up to 80% of patients trust both personal and online recommendations. 21 Because of their ease of use and accessibility, it stands to reason that online rating sites will continue to influence patient decisions in terms of choosing providers.
Because of the value that patients place on PRWs, it is important that physicians understand the impact that these platforms may have on their practices. Despite the widespread usage of PRWs by patients, surgeons remain skeptical of their utility. Additionally, some argue that it may not be a true metric to measure patient satisfaction. 3 Also, despite the objections to PRWs, they continue to have a significant impact on providers. One study found that 35% of patients selected providers based on good reports and that another 37% avoid providers based on negative reports.5,22
Given the significance of online reviews, it is important to understand the underlying factors that are associated with positive online reviews. While studies have analyzed reviews among orthopaedic sports surgeons, none, to our knowledge, has investigated the elements that contribute to an extremely positive review in orthopaedic sports medicine surgery.10,14,19,26 This is particularly relevant, given the practice pattern of sports medicine involving a large percentage of elective cases.
The purpose of this study was to analyze the factors that contribute to a 5-star review on PRWs among orthopaedic sports medicine surgeons.
Methods
Institutional review board approval was not needed for this study. Overall, 70 surgeons were randomly selected from the AOSSM Find a Doctor tool on the AOSSM website using public information. All search criteria on the Find a Doctor tool were left blank, generating a list of 3585 AOSSM-accredited surgeons. 1 A random number generator (1-3585) was utilized to determine which 70 surgeons would be selected. The number of surgeons chosen for this study was based on a prior study analyzing negative online reviews of orthopaedic sports surgeons. 20 A search was then performed for all reviews listed under the selected surgeons on Yelp.com. This website was chosen because it has been reported as the most utilized and trusted PRW by consumers. 4 All reviews other than 5 stars (of a possible 5 stars) were excluded from the study. A diagram of the inclusion and exclusion criteria is shown in Figure 1.
Figure 1.
Flowchart of inclusion and exclusion criteria for this study.
The included 5-star reviews were than classified as either surgical or nonsurgical. Reviews with a direct reference to a surgical aspect of care were assumed to be from patients who underwent surgery and were classified as surgical. Additionally, the comments within each review were classified as clinically or nonclinically related. Clinically related comments included those with a reference to good treatment results, alternative treatment methods, pain, diagnosis, or treatment plans, and nonclinical comments included physician bedside manner/professionalism, midlevel (physician assistant or nurse practitioner) professionalism, staff professionalism, wait time, time with the provider, cost, ease of billing, facilities, or ease of scheduling. The above categories were determined before analyzing the comments. All categorizations were conducted independently by 2 authors (M.E.N. and Z.L.). A third author (E.H.R.) was utilized to resolve conflicting classifications or categorizations between the initial 2 reviewers.
Univariate analysis was performed to determine means and 95% CIs. The association between categorical variables was examined with the chi-square test and Fisher exact test. An alpha value of .05 was used to determine statistical significance. The ratio of the rate of nonsurgical reviews divided by the rate of surgical reviews (rate ratio) was determined for each category and used to determine whether that category was more often being reviewed by a surgical or nonsurgical patient. In addition, the interrater reliability of the 2 reviewers was calculated. Statistical analysis was performed using Excel (Microsoft) version 16.73.
Results
A total of 1029 reviews for 70 surgeons were selected from Yelp. Of the initial 1029 reviews, 581 (57%) were identified as 5-star reviews, with the remaining excluded. Another 181 were excluded, as they were unable to be classified as 5-star reviews with either no comments or only short comments (eg, “good”). Finally, 400 five-star reviews were included for analysis, of which 200 (50%) were from surgically treated patients and 200 (50%) were from nonsurgically treated patients. These reviews were found to have 1225 positive comments, or “compliments,” with each review having an mean of 3.06 compliments. Of these compliments, 505 (41%) were clinically related, and 720 (59%) were nonclinical. The interrater reliability of the categorizations indicated 96.5% agreement, with a Cohen kappa statistic of 0.667. There was no statistically significant difference in the types of comments in terms of a clinical or nonclinical nature between the surgically and nonsurgically treated patients (Table 1).
Table 1.
Total Number and Mean Per Review of Compliments
| Overall (N = 400) | Surgical Reviews (n = 200) | Nonsurgical Reviews (n = 200) | Rate Ratio a | P | |
|---|---|---|---|---|---|
| Clinical compliments | .177 | ||||
| Total No. | 505 | 295 | 210 | 0.71 | |
| Mean per review | 1.26 | 1.48 | 1.05 | ||
| Nonclinical compliments | .072 | ||||
| Total No. | 720 | 346 | 374 | 1.08 | |
| Mean per review | 1.80 | 1.73 | 1.87 | ||
| Overall | — | ||||
| Total No. | 1225 | 641 | 584 | 0.91 | |
| Mean per review | 3.06 | 3.21 | 2.92 |
a Nonsurgical/surgical reviews.Dash indicates unable to calculate p value.
The most common positive clinical comments were for clear treatment plans (191 reviews [48%]), good outcomes (173 reviews [43%]), and providing alternative treatment options to surgery (55 reviews [14%]) (Figure 2A). The most common positive nonclinical comments were in relation to favorable physician bedside manner (287 reviews [72%]), friendly/professional staff (194 reviews [49%]), and ease of scheduling that contributed to no delays in care (68 reviews [17%]) (Figure 2B). The total number of compliments was larger than the number of reviews, as each review could contain multiple categorized comments (Table 2).
Figure 2.
Types of (A) clinical and (B) nonclinical compliments of orthopaedic sports medicine surgeons on Yelp.com.
Table 2.
Types of Comments for Compliments a
| Surgical Reviews (n = 200) | Nonsurgical Reviews (n = 200) | Rate Ratio b | P | |
|---|---|---|---|---|
| Clinical compliments | ||||
| Good outcomes | 131 (66) | 42 (21) | 0.32 | <.001 |
| Provided alternative treatment plans to surgery | 15 (8) | 40 (20) | 2.67 | <.001 |
| Well-controlled pain | 41 (21) | 11 (6) | 0.27 | <.001 |
| Correct diagnosis | 10 (5) | 15 (8) | 1.50 | .197 |
| Agree with clinical decisions/treatment plan | 1 (1) | 2 (1) | 2.00 | .480 |
| Clear treatment plans | 91 (46) | 100 (50) | 1.10 | .368 |
| No delays in care | 6 (3) | 0 (0) | — | — |
| Nonclinical compliments | ||||
| Good physician bedside manner | 141 (71) | 146 (73) | 1.04 | .679 |
| Good midlevel bedside manner | 20 (10) | 14 (7) | 0.70 | .109 |
| Friendly/professional staff | 95 (48) | 99 (50) | 1.04 | .688 |
| Short wait time | 16 (8) | 34 (17) | 2.13 | .002 |
| Enough time spent with provider | 21 (11) | 24 (12) | 1.14 | .540 |
| Cost | 0 (0) | 2 (1) | — | — |
| Clear billing/insurance | 12 (6) | 3 (2) | 0.25 | <.001 |
| Nice facilities | 8 (4) | 17 (9) | 2.13 | .029 |
| Ease of scheduling that contributed to no delays in care | 33 (17) | 35 (18) | 1.06 | .735 |
a Data are reported as n (%) unless otherwise indicated. Boldface P values indicate a statistically significant difference between groups (P < .05).Dashes indicate unable to calculate the p value.
b Nonsurgical/surgical reviews.
The most common positive clinical comments from surgically treated patients were in regard to good outcomes (131 reviews [66%]), clear treatment plans (91 reviews [46%]), and well-controlled pain (41 reviews [21%]). The most common positive nonclinical comments from surgical patients were related to good bedside manner of the physician (141 reviews [71%]), friendly/professional staff (95 reviews [48%]), ease of scheduling that contributed to no delays in care (33 reviews [17%]), and enough time spent with the provider (21 reviews [11%]).
The most common positive clinical comments from nonsurgically treated patients were for clear treatment plans (100 reviews [50%]), good outcomes (42 reviews [21%]), and providing alternative treatment options to surgery (40 reviews [20%]). The most common positive nonclinical comments from nonsurgical patients were for good bedside manner of the physician (146 reviews [73%]), friendly/professional staff (99 reviews [50%]), and ease of scheduling that contributed to no delays in care (35 reviews [18%]) (Figure 3).
Figure 3.
Comparison of compliments according to the review type.
The difference in the number of compliments for surgically treated and nonsurgical patients was statistically significant (P < .05) for good outcomes (131 vs 42 reviews, respectively), providing alternative treatment options to surgery (15 vs 40 reviews, respectively), well-controlled pain (41 vs 11 reviews, respectively), short wait times (16 vs 34 reviews, respectively), clear billing and insurance practices (12 vs 3 reviews, respectively), and impressive facilities (8 vs 17 reviews, respectively) (Table 2).
Discussion
The intent of this study was to investigate the attributes of a patient’s episode of care that led to positive reviews of orthopaedic sports medicine surgeons. Nonclinical aspects of care were more frequently commented on than clinical aspects (59% vs 41%, respectively). Nonclinical compliments most frequently referred to bedside manner, professional/friendly staff, and ease of scheduling, while clinical compliments focused on clear treatment plans, good outcomes, and providing alternative treatment options. Surgical patients tended to focus on good outcomes and control of pain, whereas nonsurgical patients commented on alternative treatment options and short wait times.
In previous studies, positive reviews tended to relate to the nonclinical aspects of patient care such as staff, convenience of the location, and the patient-physician relationship.8,11,16 Conversely, poor staff interactions and physician bedside manner tended to produce negative reviews.13,15,20,26 Furthermore, excellent bedside manner was one of the most commented on factors in 5-star reviews of physicians. It appears that patients may depend on nonclinical aspects of care for evaluating overall quality in the medical field. 18 Therefore, when hiring staff or interacting with patients, it is important that surgeons take into consideration the way that staff members and their behaviors are perceived.
Even though patients focus on nonclinical aspects of care, having friendly, accommodating staff and good bedside manners are not the only characteristics seen in positive reviews. Perceived technical competence has previously been reported as a common factor in higher physician ratings.16,19 In our study, 5-star ratings referenced clinical skills in 41% of the compliments, most frequently focusing on clear treatment plans and good outcomes. While previous studies have argued that PRWs have a poor correlation with a physician’s clinical capability, our study finds that patients still placed a high value on favorable clinical outcomes.6,22,23 However, it is important to note that PRWs are inherently subjective and typically do not objectively reflect health care quality.
Additionally, orthopaedic surgeons need to be aware of any differences between reviews left by surgical versus nonsurgical patients. For negative reviews, 1 study showed that nonsurgical patients were more likely to leave poor reviews than surgical patients. 20 Furthermore, surgical patients focused on their surgical course, with comments pertaining to complications, reoperations, and readmissions compared to nonsurgical patients, who focused on unclear treatment plans and misdiagnoses. 20 Both surgical and nonsurgical patients emphasized uncontrolled pain, delays in care, and bedside manner in negative reviews. For positive reviews, there were a few categories that surgical patients complimented more frequently than nonsurgical patients including good outcomes, well-controlled pain, and clear billing. Nonsurgical patients had more comments pertaining to alternative treatment plans to surgery and short wait times. Both surgical and nonsurgical patients focused on good physician bedside manner, clear treatment plans, and friendly staff. Thus, orthopaedic sports medicine surgeons should be cognizant of their interactions with patients, communication of treatment plans, and controlling pain, with acknowledgment that surgical and nonsurgical patients are equally likely to leave positive reviews.
Previous reports have expressed concerns about the legitimacy of online reviews and the possibility of false reviews tarnishing a physician’s practice and reputation. 18 Although Yelp was selected for its reputation as the most utilized and trusted PRW by consumers, up to 20% of its reviews have been shown to be fraudulent.4,26 In fact, Lagu et al 13 were able to identify reviews written by physicians themselves when investigating PRWs. However, Yelp.com does offer software that will identify and hide reviews that were solicited by the business, but it does not prevent fraudulent reviews. 25 As physician reputation is becoming increasingly considered by patients, there is a financial incentive for providers to increase their online rating. Additionally, there are reports of companies dedicated to posting positive reviews as well as fake negative reviews, making it even more difficult to maintain the validity of PRWs. 7 Future studies could assess the validity of online reviews and propose ways to ensure their integrity moving forward. One such way would be to compare online hospital scores (verified) versus PRW online scores. With the future increased trustworthiness of PRWs, physicians would have confidence that their efforts to provide excellent care, practice good bedside manner, and promote a friendly staff environment will be rewarded with extremely positive reviews.
Limitations
There were several limitations to this study. Selection bias may be present, as we limited our online review platform to Yelp.com. This may not accurately reflect the components of positive reviews if other platforms are visited by different patients. We attempted to minimize this issue by using a platform that is the most utilized and trusted by consumers. 4 Selection bias may also be present because we chose 70 orthopaedic sports medicine surgeons of a possible 3585, potentially limiting the external validity of this study. By using a randomized generator, we hoped to increase our relatability and reduce our impact on external validity.
Conclusion
The majority of 5-star reviews complimented nonclinical aspects of care such as physician bedside manner and friendly staff. The most common clinical compliments were related to good outcomes and clear treatment plans. The overall most common compliment referred to good bedside manner from both surgical and nonsurgical patients.
Footnotes
Final revision submitted February 22, 2023; accepted March 20, 2023.
One or more of the authors has declared the following potential conflict of interest or source of funding: S.T. has received education payments from ImpactOrtho. A.C. has received education payments from Arthrex and consulting fees from Zimmer Biomet. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.
References
- 1.American Orthopaedic Society for Sports Medicine. Sports medicine doctors. Accessed December 15, 2021. https://www.sportsmed.org/AOSSMIMIS/Members/Applications/Find_A_Doctor.aspx
- 2.Bernstein DN, Mesfin A. Physician-review websites in orthopaedic surgery. JBJS Rev. 2020;8(3):e0158. [DOI] [PubMed] [Google Scholar]
- 3.Chen J, Presson A, Zhang C, Ray D, Finlayson S, Glasgow R. Online physician review websites poorly correlate to a validated metric of patient satisfaction. J Surg Res. 2018;227:1–6. [DOI] [PubMed] [Google Scholar]
- 4.Chowdhury A. What are the top doctor rating and review sites? Doctible. Published 2017. Accessed January 3, 2022. https://blog.doctible.com/what-are-the-top-doctor-rating-and-review-sites-20d32ba1ec81
- 5.Donnally CJ, McCormick JR, Li DJ, et al. How do physician demographics, training, social media usage, online presence, and wait times influence online physician review scores for spine surgeons? J Neurosurg Spine. 2019;30:279–288. [DOI] [PubMed] [Google Scholar]
- 6.Emmert M, Sander U, Esslinger AS, Maryschok M, Schöffski O. Public reporting in Germany: the content of physician rating websites. Methods Inf Med. 2012;51(2):112–120. [DOI] [PubMed] [Google Scholar]
- 7.Flores C. Businesses use fake online reviews to scam customers. CBS Austin. Published March 15, 2022. Accessed March 22, 2022. https://cbsaustin.com/news/local/businesses-use-fake-online-reviews-to-scam-customers
- 8.Gao GG, McCullough JS, Agarwal R, Jha AK. A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res. 2012;14(1):e38. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014;311(7):734–735. [DOI] [PubMed] [Google Scholar]
- 10.Heimdal TR, Gardner SS, Dhanani UM, Harris JD, Liberman SR, McCulloch PC. Factors affecting orthopedic sports medicine surgeons’ online reputation. Orthopedics. 2021;44(2):e281–e286. [DOI] [PubMed] [Google Scholar]
- 11.Kadry B, Chu LF, Kadry B, Gammas D, MacArio A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011;13(4):e95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kaiser Family Foundation. 2008 update on consumers’ views of patient safety and quality information. Published September 30, 2008. Accessed January 19, 2022. https://www.kff.org/health-reform/poll-finding/2008-update-on-consumers-views-of-patient-2/
- 13.Lagu T, Hannon NS, Rothberg MB, Lindenauer PK. Patients’ evaluations of health care providers in the era of social networking: an analysis of physician-rating websites. J Gen Intern Med. 2010;25(9):942–946. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Langerhuizen DWG, Brown LE, Doornberg JN, Ring D, Kerkhoffs GMMJ, Janssen SJ. Analysis of online reviews of orthopaedic surgeons and orthopaedic practices using natural language processing. J Am Acad Orthop Surg. 2021;29(8):337–344. [DOI] [PubMed] [Google Scholar]
- 15.Liu C, Uffenheimer M, Nasseri Y, Cohen J, Ellenhorn J. “But his Yelp reviews are awful!”: analysis of general surgeons’ Yelp reviews. J Med Internet Res. 2019;21(4):e11646. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.López A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012;27(6):685–692. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Mehta SJ. Patient satisfaction reporting and its implications for patient care. AMA J Ethics. 2015;17(7):616–621. [DOI] [PubMed] [Google Scholar]
- 18.Murphy GP, Radadia KD, Breyer BN. Online physician reviews: is there a place for them? Risk Manag Healthc Policy. 2019;12:85–89. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Nwachukwu BU, Adjei J, Trehan SK, et al. Rating a sports medicine surgeon’s “quality” in the modern era: an analysis of popular physician online rating websites. HSS J. 2016;12(3):272–277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Pollock JR, Arthur JR, Smith JF, et al. The majority of complaints about orthopedic sports surgeons on Yelp are nonclinical. Arthrosc Sports Med Rehabil. 2021;3(5):e1465–e1472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Qiu CS, Hockney SM, Turin SY, Dorfman RG, Kim JYS. A quantitative analysis of online plastic surgeon reviews for abdominoplasty. Plast Reconstr Surg. 2019;143(3):734–742. [DOI] [PubMed] [Google Scholar]
- 22.Sequist TD, Schneider EC, Anastario M, et al. Quality monitoring of physicians: linking patients’ experiences of care to clinical quality and outcomes. J Gen Intern Med. 2008;23(11):1784–1790. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Trehan SK, Nguyen JT, Marx R, et al. Online patient ratings are not correlated with total knee replacement surgeon–specific outcomes. HSS J. 2018;14(2):177–180. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.US Department of Health and Human Services. HHS Protect Public Data Hub. Accessed March 22, 2022. https://protect-public.hhs.gov/
- 25.Yelp.com Support Center. Don’t ask for reviews. Accessed January 24, 2022. https://www.yelp-support.com/article/Don-t-Ask-for-Reviews?l=en_US
- 26.Yu J, Samuel LT, Yalçin S, Sultan AA, Kamath AF. Patient-recorded physician ratings: what can we learn from 11,527 online reviews of orthopedic surgeons? J Arthroplasty. 2020;35(6):S364–S367. [DOI] [PubMed] [Google Scholar]
- 27.Zeckhauser R, Sommers B. Consumerism in health care: challenges and opportunities. Virtual Mentor. 2013;15(11):988–992. [DOI] [PubMed] [Google Scholar]