Abstract
Background: Physician review websites are now commonly used by patients. However, in facial plastic surgery, the trends and content in these websites are not well studied. We examined online reviews for U.S. facial plastic surgeons, and compared comment content with the most commonly used patient experience survey, the Consumer Assessment of Healthcare Providers and Systems (CAHPS) administered by Press Ganey.
Methods: A retrospective mixed method study was employed to quantitatively compare online ratings and comments of 100 randomly selected U.S. facial plastic surgeons on vitals.com, healthgrades.com, google.com and zocdoc.com. Qualitative content analysis was utilized to categorize themes present in 957 patient-generated (unverified) comments, and compare these with CAHPS survey questions and themes.
Results: The physician review websites had favorable ratings of facial plastic surgeons with 84.55% five-star reviews on Healthgrades and 78.40% on Vitals. These ratings were similar across surgeon age (p = 0.44), gender (p = 0.85), and geographic region (p = 0.29). Of sites examined, Healthgrades and Vitals were most frequently used. Analysis of patient comments identified themes aligning with CAHPS content (e.g., physician interactions, efficiency, and recommendation likelihood), as well as additional themes such as patient's outcome perception (55.28% of comments) and finances (86% of negatively rated reviews).
Conclusions: These exploratory results suggest that facial plastic surgeons are generally rated positively online, and the comments left on these websites provide additional feedback that is not currently included in CAHPS surveys. In evaluating the patient experience with facial plastic surgery practices, these websites may prove to be useful.
Key Points
Question: What do patients say about facial plastic surgeons on physician review websites, and do these reviews supplement the content of Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys?
Findings: These exploratory results demonstrate that most online ratings are positive, and do not appear to be influenced by surgeon demographics. Reviewer comments provide additional insight into patient experiences that supplement CAHPS, such as their perceptions of outcomes and care costs.
Meaning: Physician review websites provide additional information regarding facial plastic surgery patient experiences that are not available in CAHPS surveys and represent potentially unique feedback regarding care delivered.
Introduction
Increasingly, patients are using online physician rating websites (PRWs) before selecting their physician.1 Approximately 35% of patients report seeing a physician because of positive online ratings, whereas a similar proportion of patients (37%) reported that negative reviews led them to seek care elsewhere.2 In facial plastic surgery, PRWs are particularly important3 as procedures are often elective, cosmetic, and paid for out of pocket. Similarly, PRW use is highest in female patients and in those with an undergraduate degree,1 which aligns with facial plastic surgery populations.
With increased patient use of PRWs, maintaining an online reputation is an important component of practice management. Examples of strategies employed include hiring firms to manage PRWs, requiring patients to sign waivers prohibiting writing online reviews,4 and litigation of patients for libel.5 Although PRWs are widely used, the quality of data provided is unclear, as many PRWs do not verify that comments are generated by patients. Despite these negative connotations, there is evidence that overall, patients rate physicians favorably,6 especially plastic surgery patients.7 In addition, even with the questions regarding review validity, it is important to study all information patients are using to make health care decisions.
As the United States transitions toward a value-based health care system, there is increased focus on measuring quality and patient experience. Each specialty within medicine is inherently different, and it is rare to find a comprehensive quality indicator. Thus, at the moment, patient experience surveys are widely used as a proxy for assessing quality of care delivered.
The Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys were developed by The Agency for Healthcare Quality and Research (AHRQ) to capture patient experiences for a wide variety of care settings.8 The CAHPS surveys are the most commonly used experience surveys for insurance-based care and are typically administered by a third-party vendor such as Press Ganey Associates, a private company.9 In this way, patient experience may be measured and reported to the Centers for Medicare & Medicaid Services (CMS). CMS, in turn, uses Quality Payment Programs such as the Merit-Based Incentive Payment System (MIPS) and Alternative Payment Models (APMs), which use the CAHPS surveys to determine quality and these are linked to reimbursement.10 Eventually, CAHPS scores are expected to be published publicly on the CMS Physician Compare website.11,*
Patient experience data (e.g., CAHPS surveys and PRW reviews) play at least three critical roles in our current system: (1) to inform patients during provider selection, (2) as a feedback tool to clinicians, and (3) as a potential performance indicator to third parties. Evidence examining PRWs of hospitals indicates that reviews provided additional important quality measure domains not captured by CAHPS surveys.12 However, it is unknown how the surveys used administered by Press Ganey (Outpatient and Ambulatory Surgery CAHPS, OAS-CAHPS; and Surgical Care CAHPS, S-CAHPS) compares with common content from PRWs for facial plastic surgeons. Facial plastic surgery is a unique field where a significant portion of care is provided through a self-pay model and thus examining patient experience tools outside of the insurance-based CAHPS surveys, such as PRWs, becomes increasingly relevant.
Methods
Design
This retrospective concurrent mixed method study explores how PRWs are used to evaluate performance of facial plastic surgeons.13 Specifically, using a national sample of facial plastic surgeons (n = 100), the number of reviews and comments were quantitatively compared across four commonly used PRWs. Then, the relationship between surgeon characteristics and overall ratings for individual surgeons was determined for the two most frequently used PRWs. Finally, a purposeful sample of narrative comments was qualitatively analyzed to describe patient experiences with facial plastic surgeons and to compare with CAHPS content. This study was deemed exempt by the Institutional Review Board of Thomas Jefferson University.
Sample
The American Academy of Facial Plastic and Reconstructive Surgery (AAFPRS) membership directory was used to construct a random sample of 100 facial plastic surgeons.14 The random sample was constructed by selecting every 13th individual from the publicly available directory. To be included, the physician needed to practice in the United States, be board-certified by the American Board of Otolaryngology and/or the American Board of Facial Plastic and Reconstructive Surgery, and have >1 year of postfellowship experience.
The sample included data from 2015 to 2019 from individual physicians' PRW pages; PRWs at the hospital- or practice-level were excluded. For qualitative analysis, a purposeful sample13 of Healthgrades comments was used. Healthgrades was selected because it is a frequently used PRW with an open-comment format and limited comment censorship.15 To minimize bias, analysis was limited to 40 comments per surgeon; therefore, if applicable, the 20 most recent lowest/highest rated comments were included.
Data collection
In May 2019, an online search was conducted to obtain each surgeon's publicly available data from healthgrades.com, vitals.com, google.com, and zocdoc.com. Surgeon demographics and comments were collected from Healthgrades.com, whereas the number of reviews/comments and star-rating scores were collected from all sites.
Variables
Overall ratings
All platforms use a one- to five-star Likert scale and use an average of reviewers' scores to determine the overall rating. However, the PRWs vary in overall rating content focus. Healthgrades15 and ZocDoc16 overall scores reflect likelihood to recommend, whereas Google17 and Vitals18 reflect patient's overall provider experience.
Surgeon characteristics
Healthgrades was used to identify surgeon demographics: age, gender, years of postfellowship experience, and location. Quarterly, Healthgrades determines information from public databases.19 U.S. census definitions of geographic regions were used.20
Engagement
Two measures of patient engagement were determined for each PRW: number of reviews and comments per surgeon.
Analysis
All statistical analyses were conducted using Stata/IC statistical software (release 15; StataCorp LP). Moody's median and Kruskal–Wallis tests were used to compare comment/review numbers with geographic region. Kruskal–Wallis tests were employed to determine the relationship between surgeon characteristics and overall star ratings for individual surgeons for the most frequently used PRWs.
Content analysis13 of narrative review comments was used to describe patient experiences with facial plastic surgeons. Themes were identified both inductively from reviewer comments, and deductively from the CAHPS questions. First, two researchers (C.D.B. and K.A.M.) independently open-coded comments to identify author-generated inductive themes. Word clouds of comments with the highest/lowest rated reviews were used to illustrate linguistic patterns. Then, using a codebook, reviewer comments were independently recoded to examine theme prevalence and to identify representative quotes. Coder interrater reliability was high (Kappa = 0.89).
Results
Sample
A random sample of 100 American facial plastic surgeons was determined from ∼1300 eligible providers. Table 1 provides sample demographic data. The sample was predominantly male (n = 87), and the average age was 51 (standard deviation = 9.6). Of the 100 physicians examined, 95 had at least one review posted on Healthgrades, compared with 90 on Vitals, 68 on Google, and 16 on ZocDoc. However, in general, online PRW ratings are most reliable when there are at least five ratings available.21 In our sample, 74 physicians had a reliable rating (≥5 reviews) on Healthgrades, compared with 75 on Vitals, 38 on Google, and 11 on ZocDoc.
Table 1.
Demographic data of facial plastic surgeons sampled (n = 100)
| Characteristic | Category | Count |
|---|---|---|
| Gender | Male | 87 |
| Female | 13 | |
| Geographic region | South | 33 |
| East | 19 | |
| Midwest | 15 | |
| West | 33 | |
| Age (years) | 30–40 | 11 |
| 40–50 | 32 | |
| 50–60 | 38 | |
| >60 | 19 | |
| Experience (years) | 0–10 | 17 |
| 11–20 | 33 | |
| 21–30 | 37 | |
| >30 | 13 |
Patient engagement
Table 2 contains data regarding patient engagement across PRWs. The PRW with the highest number of reviews was Vitals (n = 4129 reviews), followed by Healthgrades (n = 3643), Google (n = 1492), and ZocDoc (n = 1292).
Table 2.
Patient engagement with physician review websites
| Platform type | Engagement metric | Total (n) | Mean (SD) per surgeon | Median per surgeon | Overall min/max per surgeon |
|---|---|---|---|---|---|
| All sites | Reviews | 10,556 | 105.56 (76.91) | 5.00 | 0/906 |
| Comments | 5610 | 56.10a (48.51) | 2.00 | 0/760a | |
| Healthgrades | Reviews | 3643 | 36.43 (71.13) | 13.00 | 0/499 |
| Commentsa | 1305 | 13.05 (19.47) | 3.00 | 0/75 | |
| Vitals | Reviews | 4129 | 41.29 (93.93) | 15.50 | 0/778 |
| Comments | 2620 | 26.20 (83.44) | 6.50 | 0/760 | |
| Reviews | 1492 | 14.92 (28.44) | 3.00 | 0/182 | |
| Comments | 1133 | 11.33 (24.04) | 2.00 | 0/161 | |
| ZocDoc | Reviews | 1292 | 12.92 (92.29) | 0.00 | 0/906 |
| Comments | 552 | 5.52 (36.48) | 0.00 | 0/353 |
Per website protocols the Healthgrades number of comments are limited to a maximum of 75 per surgeon. There were four surgeons with >75 comments.
The median number of reviews varied by geographic region for Healthgrades (χ2 = 8.24, p = 0.041): median number of reviews was higher for surgeons working on the east coast compared with the south (χ2 = 8.86; p = 0.003) or west coast (χ2 = 9.71; p = 0.002), and was higher in the midwest than on the west coast (χ 2 = 5.47, p = 0.02). No variation in the median number of reviews per surgeon across geographic regions was found for Vitals (χ2 = 8.86; p = 0.003), Google (χ2 = 1.62; p = 0.68), or ZocDoc (χ2 = 2.96; p = 0.42). Appendix Tables A1–A3 provides geographic data.
Performance ratings
All four PRWs showed similarly favorably skewed distributions of review ratings. Figure 1 provides the rating distributions for the top two PRWs: Healthgrades and Vitals. For Healthgrades, 11.14% were one star and 84.55% were five star (mean 4.48 stars). Similarly, for Vitals, 9.76% were one star and 78.40% were five star (mean 4.40 stars). There were disproportionately more five-star and one-star reviews on both sites compared with two-, three-, and four-star ratings (χ2 = 284.11, p < 0.001).
Fig. 1.
Distribution of overall star review scores of facial plastic surgeons on Healthgrades (n = 3643 reviews) and Vitals (n = 4129 reviews). Healthgrades overall scores reflect likelihood to recommend; Vitals reflect patient's overall experience with the provider.
For Healthgrades, individual ratings did not differ by surgeon characteristics: gender (χ2 = 0.03; p = 0.85), geographic region (χ2 = 4.01; p = 0.26), age (χ2 = 2.68; p = 0.44), or years of experience (χ2 = 2.77; p = 0.26). The results for Vitals were similar (data not shown). Appendix Tables A4–A6 provides additional surgeon demographic data.
Narrative review themes
The qualitative sample included 957 narrative review comments for 74 facial plastic surgeons on Healthgrades. The average number of reviewer comments per surgeon was 12.6 (median = 8; min/max: 1/40). The narrative reviews included 125 one-star reviews (13.06%), 16 two-star reviews (1.67%), 3 three-star reviews (0.31%), 19 four-star reviews (1.99%), and 794 five-star reviews (82.97%). Figure 2 displays word clouds of the most commonly used words in comments with one-star (n = 125) and five-star (n = 794) reviews. Examples of frequently used words for one-star reviews included “wait,” “told,” “rude,” and “never” (as in “never going back”). Examples of common words unique for five-star reviews included “results,” “feel,” “happy,” “look,” and “caring.”
Fig. 2.
Word clouds illustrating linguistic patterns of comments with the lowest (n = 125) and highest (n = 794) star-rated reviews. The letter size and darkness increase with frequency of word use. (A) One-star reviews. (B) Five-star reviews.
CAHPS questions thematic representation
The content of the OAS-CAHPS and S-CAHPS surveys was sorted into CAHPS themes: overall experience, recommendation, physician interactions, staff interactions, efficiency, check-in, preprocedure, anesthesia instructions, discharge instructions, cleanliness, pain, bleeding, infection, and nausea/vomiting. Table 3 provides the prevalence of themes in reviewers' comments. Appendix Tables A7 and A8 contain each theme, associated survey questions, and examples of positive/negative representative reviewer quotes.
Table 3.
Presence and tone of thematic trends in narrative reviewer comments from Healthgrades for facial plastic surgeons (n = 957 reviewer comments)
| Source | Theme | Presence, % (n) | Tone of theme |
|
|---|---|---|---|---|
| Positive, % (n) | Negative, % (n) | |||
| CAHPS | Overall experience | 99.27 (950) | 84.63 (804) | 15.37 (146) |
| Physician interactions | 58.90 (564) | 87.59 (494) | 12.41 (70) | |
| Recommendation | 31.45 (301) | 84.39 (254) | 15.61 (47) | |
| Staff interactions | 30.72 (294) | 89.12 (262) | 10.88 (32) | |
| Time/efficiency | 19.85 (190) | 68.95 (131) | 31.05 (59) | |
| Preprocedure information | 13.48 (129) | 86.05 (111) | 13.95 (18) | |
| Discharge instructions | 5.85 (56) | 82.14 (46) | 17.86 (10) | |
| Check-in | 5.64 (54) | 62.96 (34) | 37.04 (20) | |
| Pain | 4.39 (42) | 73.81 (31) | 26.19 (11) | |
| Cleanliness | 1.46 (14) | 100 (14) | 0 (0) | |
| Bleeding | 0.31 (3) | 33.33 (1) | 66.66 (2) | |
| Infection | 0.21 (2) | 0 (0) | 100 (2) | |
| Anesthesia instructions | 0.11 (1) | 100 (1) | 0 (0) | |
| Nausea/vomiting | 0 (0) | N/A | N/A | |
| Author-inductive | Bedside manner | 58.62 (605) | 82.48 (499) | 10.25 (62) |
| Overall outcome | 55.28 (529) | 86.77 (459) | 13.23 (70) | |
| Answered questions | 31.24 (299) | 89.63 (268) | 10.37 (31) | |
| Cosmetic outcome | 30.62 (293) | 88.05 (258) | 11.95 (35) | |
| Follow-up | 18.39 (176) | 78.98 (139) | 21.02 (37) | |
| Functional outcome | 15.36 (147) | 78.91 (116) | 21.09 (31) | |
| Knowledge | 14.63 (140) | 86.43 (121) | 13.57 (19) | |
| Finances | 5.22 (50) | 14.00 (7) | 86.00 (43) | |
| Personal communication | 3.66 (35) | 94.29 (33) | 5.71 (2) | |
CAHPS, Consumer Assessment of Healthcare Providers and Systems Survey; N/A, not available.
The most common theme was overall experience, which was present in 99.27% of comments (n = 950) and, of these, 84.63% reflected a positive overall experience (n = 804). 31.45% of comments discussed likelihood to recommend the physician. Similarly, likelihood to recommend was positive in 87.59% of comments addressing this theme. The physician interactions theme encompassed the most CAHPS questions compared with the other themes analyzed (Appendix Tables A7, A8), and, unsurprisingly, was the second most common theme patients specifically commented on through PRWs (present in 58.90% of comments).
Themes that, when addressed, were most commonly portrayed in a positive manner were overall experience, physician interactions, recommendation, staff interactions, preprocedure information, discharge instructions, cleanliness, and anesthesia instructions. In contrast, efficiency, check-in, pain, bleeding, and infection were themes more likely to be associated with negative reviews. Notably, comments on pain, cleanliness, bleeding, infection, anesthesia instructions, and nausea/vomiting were present in <5% of the comments.
Additional themes
Nine inductive author-generated themes were identified from the reviewer comments, which may not be reflected by the CAHPS surveys. These include bedside manner, outcome (cosmetic and functional), answered questions, follow-up, knowledge, finances, and personal communication.
The theme bedside manner describes the physician's approach/attitude. Generally, this theme was expressed as the patient commenting on the personality traits and communication skills of the physician. Bedside manner was present in 58.62% of comments, and was similar in prevalence and content to the physician interactions theme generated by CAHPS questions (58.90% of comments). The theme of answered questions also seemed to be valued, as almost one-third of reviewers commented on this theme (31.24%, n = 299), typically in a positive manner (89.63%, n = 268). In addition, surgeon's knowledge was mentioned in 14.64% of comments and reviewers frequently described their facial plastic surgeon being “intelligent,” “up to date,” and “confident.”
Overall outcome was present in 55.28% of comments (n = 529) and was the second most common additional theme, and fourth most common theme overall. This theme captured cosmetic, functional, and nonspecified outcomes. Cosmetic outcome was described in 30.62% of comments (n = 293) and was more common than functional outcomes (15.36%, n = 147). Cosmetic outcome-related comments routinely described patients having increased “confidence,” a “natural” appearance, and willingness to recommend to others. Functional outcome comments were associated with descriptions of “improved breathing” from functional rhinoplasty, as well as comments about general otolaryngological procedures performed by facial plastic surgeons such as endoscopic sinus surgery and tympanostomy tube placement.
The follow-up theme described patients' perceptions around continuity of care with their facial plastic surgeon. In total, 176 reviewers (18.39%) commented on follow-up, which predominantly focused on how they felt taken care of during the surgical recovery period. Negative follow-up reviews tended to have poor communication experiences with the office and/or were not provided with information about when/where to seek care if a problem arose. The theme of personal communication is related to the follow-up theme, which was defined as the physician directly communicating with patients, outside of the office or hospital. The comments in this theme described their facial plastic surgeons providing their personal cell phone number, e-mail address, or physician-initiated personal communication in the immediate postoperative period.
Finances was the sole negative theme described in the additional themes section. Although uncommon (5.22%, n = 50), comments were overwhelmingly negative (86.00%, n = 43) and were associated with a negative outcome where patients felt like they had “wasted money.” The few comments describing positive experiences mentioned how the “office helped work with their budget” or how their procedure was “worth the cost.”
Discussion
This mixed methods study explores nationwide trends across physician review websites for facial plastic surgeons, and compares content of patient comments on these websites with themes found in the CAHPS surveys. Of the PRWs studied, Healthgrades and Vitals were most commonly used, which provide similar ratings regardless of surgeon age, experience, gender, or location. These two PRWs were also found to be most common in general plastic surgeons7 and otolaryngologists.3 The only other study on PRWs specifically for facial plastic surgeons focused on Yelp reviews in five large U.S. cities.22 However, in our preliminary analysis examining which websites to include, Yelp seemed to be present for fewer facial plastic surgeons and data for individual surgeons were frequently unavailable. This study adds to the literature by examining multiple PRWs, comparing content between PRWs and CAHPS surveys, and using a randomly selected sample of surgeons from a range of urban, suburban, and rural areas.
The overall five-star review ratings for facial plastic surgeons in this study was 84.55% on Healthgrades and 78.40% on Vitals, which is similar to the rates observed in general plastic surgery.7 The average Healthgrades rating of facial plastic surgeons was 4.48, which is higher than the reported value of 3.97 reported in the study by Sobin and colleagues—which was limited to facial plastic surgeons in a northeast academic setting.3 In that same study, there was a larger proportion of negative comments observed (31.6%), which is almost double the frequency seen in our study (15.37%). This discrepancy may be due to poorer patient satisfaction seen in academic institutions as one study comparing otolaryngologists' posits.23
The bimodal distribution of ratings skewed toward either five stars or one star is a recognized quality for PRW ratings across subspecialties.24 This may be due to a certain threshold of patient emotional involvement influencing their decision to make a conscious effort to seek out a PRW and leave an online review. In any case, examining negative and positive comments may provide both patients and providers with valuable information in real time. Positive comments may provide insight into what is going well within a practice. Negative comments, although a source of physician frustration, are an accessible avenue of transparency in the health care system where patients may keep providers and their staff accountable. Interestingly, comments from PRWs for emergency departments share similar linguistic patterns for negative reviews as demonstrated in this study—as “told,” “rude,” and “never” were common in both populations.25 Positive reviews were more focused on “helpfulness” and being “friendly” in emergency department reviews, compared with “results,” “looking,” and “feeling” for facial plastic surgeon reviews.25
When evaluating prevalence between CAHPS-generated themes and author-generated themes, our findings identified important aspects of quality patient experiences not captured in CAHPS surveys.12 Namely, patient perceptions of outcomes emerged as an important theme from patient comments on PRWs, but are not touched on in CAHPS surveys. It is interesting that the subjective experience captured by patients in these surveys omits their interpretation of whether their treatment or surgery was successful. As supported by our findings, this is arguably one of the most important factors patients look for when choosing a facial plastic surgeon, hence the importance of before and after photos and the rise of professional Instagram pages for facial plastic surgery practices. In contrast, several CAHPS themes had a low prevalence in review comments. The lack of comments may indicate the qualities represented by those themes are either expected by most patients, unimportant to most patients, or present in a very low frequency.
Limitations
Data obtained from PRWs are not without limitations. One cannot directly compare ratings across PRWs, as they are inherently using different criteria to determine “5-star ratings.” The prevalence of fake reviews on PRWs is unknown, and may skew data in these open online settings.26,27 Despite the questionable validity of PRW patient reviews/comments, it is important to take them all into context as these are what patients see and use to make decisions about health care.12 In addition, patients who undergo care outside of insurance plans currently do not have CAHPS surveys to express their experiences, and PRWs offer an accessible avenue to relay feedback.
For quality payment programs through CMS, there is the option to add additional questions to the OAS-CAHPS and S-CAHPS surveys from the Clinician and Group CAHPS survey (CG-CAHPS).8 However, these additional questions do not seem to exhibit the author-described themes found in this study. Importantly, physician-specific CAHPS scores were not available to compare with these individual physicians' online ratings; however, these measures have shown to capture different phenomena when compared in the setting of academic otolaryngologists.28 Finally, during theme generation, it is possible that others may have identified additional themes.29
Conclusions
Facial plastic surgeons are overall positively rated online and, of the websites studied, Healthgrades and Vitals appear to be the most commonly used, regardless of individual characteristics or region within the United States. The comments available on physician review websites provide important information to those seeking care, which is not encompassed in the CAHPS surveys generated by CMS and administered by Press Ganey. They also provide data from patients who would otherwise not receive CAHPS surveys. Thus, PRWs' narrative reviews provide helpful additional data to supplement traditional patient experience surveys such as CAHPS and allow self-pay patients to evaluate their experience with facial plastic surgery practices.
Appendix
Appendix Table A1. Mood's median test to examine difference in median number of reviewer comments for each physiciana review website by region
| Website | No. of reviews | Overall median | χ2 (df = 3) | p |
|---|---|---|---|---|
| Healthgrades | 3643 | 13.00 | 8.24 | 0.04* |
| Vitals | 4129 | 15.50 | 3.43 | 0.34 |
| 1492 | 3.00 | 1.62 | 0.68 | |
| ZocDoc | 1292 | 0 | 2.96 | 0.42 |
100 physicians.
Significant p-value (<0.05).
Appendix Table A2. Average and median number of Healthgrades reviews of facial plastic surgeons (n = 100) by region
| Regions compared | No. of physicians | Total no. of reviews | Mean reviews per physician | Median reviews per physician |
|---|---|---|---|---|
| East | 19 | 1088 | 57.26 | 22 |
| South | 33 | 1203 | 36.45 | 12 |
| West | 33 | 640 | 19.38 | 9 |
| Midwest | 15 | 591 | 39.4 | 22 |
Appendix Table A3. Results of follow-up Kruskal–Wallis to examine regional differences in the number of reviews on Healthgrades for 100 facial plastic surgeons
| Regions compared | χ2 (df = 1) | p |
|---|---|---|
| East-South | 8.86 | 0.003* |
| East-Midwest | 1.11 | 0.292 |
| East-West | 9.71 | 0.002* |
| West-Midwest | 5.47 | 0.020* |
| West-South | 0.78 | 0.376 |
| South-Midwest | 0.96 | 0.327 |
Significant p-value (<0.05).
Appendix Table A4. Description of star ratings for Healthgrades (n = 3522 reviews)
| Rating | Total no. (%) of reviewsa | Average no. of reviews per MD | Min | Max | SD |
|---|---|---|---|---|---|
| 1 | 406 (11.14) | 4.27 | 0 | 27 | 5.50 |
| 2 | 52 (1.42) | 0.57 | 0 | 7 | 1.05 |
| 3 | 15 (<1) | 0.17 | 0 | 3 | 0.49 |
| 4 | 90 (2.20) | 0.95 | 0 | 22 | 2.67 |
| 5 | 3080 (84.55) | 32.42 | 0 | 501 | 72.90 |
Only providers with Healthgrades review scores were used (n = 95 physicians).
MD, medical doctor; SD, standard deviation.
Appendix Table A5. Description of star ratings for Vitals (n = 4166 reviews)
| Rating | Total no. (%) of reviewsa | Average no of reviews per MD | Min | Max | SD |
|---|---|---|---|---|---|
| 1 | 403 (9.76) | 4.48 | 0 | 51 | 5.50 |
| 2 | 155 (3.72) | 1.72 | 0 | 16 | 1.05 |
| 3 | 71 (1.7) | 0.79 | 0 | 6 | 0.49 |
| 4 | 263 (6.31) | 2.92 | 0 | 35 | 2.67 |
| 5 | 3237 (78.40) | 35.97 | 0 | 728 | 72.90 |
Only providers with Vitals review scores were used (n = 90 physicians).
Appendix Table A6. Comparing reviewer ratings on Healthgradesa by gender, geographic regions, age, and experience for 74 surgeons
| Characteristic | Category | No. of physicians | χ2 | p |
|---|---|---|---|---|
| Gender | Male | 70 | 0.03 | 0.85 |
| Female | 12 | |||
| Geographic region | South | 28 | 4.01 | 0.26 |
| East | 11 | |||
| Midwest | 11 | |||
| West | 23 | |||
| Age (years) | 30–40 | 18 | 2.68 | 0.44 |
| 40–50 | 18 | |||
| 50–60 | 23 | |||
| >60 | 15 | |||
| Experience (years) | 0–10 | 18 | 3.77 | 0.29 |
| 11–20 | 19 | |||
| 21–30 | 22 | |||
| >30 | 15 |
Only providers with valid Healthgrades review scores (>5 reviews per surgeon) were used.
Appendix Table A7. Linking themes to outpatient and ambulatory surgery CAHPS and surgical care CAHPS to representative quotes from Healthgrades narrative review comments (n = 957)
| Theme | OAS and S-CAHPS questions | Positive comment example | Negative comment example |
|---|---|---|---|
| Check-in | Question (Q) 3OAS-CAHPS: Did the check-in process run smoothly? | “Making appointments have never been easy and I've never had to wait longer than a few minutes, if that. The office staff made me feel so welcome.” | “We got a call in the morning to be there at 10, and had to rush there since we were previously told to bet there at 12:30, then he wasn't seen until 3:30!” |
| Preprocedure information |
Q1-OAS: Before your procedure, did your doctor or anyone from the facility give you all the information you needed about your procedure? Q2-OAS: Before your procedure, did your doctor or anyone from the facility give you easy to understand instructions about getting ready for your procedure? Q9-OAS: Did the doctors and nurses explain your procedure in a way that was easy to understand? Q3-S: Before your surgery, did anyone in this surgeon's office give you all the information you needed about your surgery? Q4-S: Before your surgery, did anyone in this surgeon's office give you easy to understand instructions about getting ready for your surgery? |
“She took the time to explain to us the plan of care, and made sure my mom understood each step in the process. It was an exceptionally pleasant experience. We felt ready for the procedure and like we were in good hands.” “Very helpful in explaining steps of the procedure.” |
“The doctor did not inform me of any of the possible risks.” “He does a very brief evaluation and does not explain pros and cons or details about what surgery he may provide.” “I do not feel like I realized what a big surgery it was or the risks before I had it done. If I had, I'm not sure I would have gone through with it.” |
| Cleanliness | Q4-OAS: Was the facility clean? | “The office was very clean and inviting.” | N/A |
| Anesthesia instructions |
Q10-OAS: Anesthesia is something that would make you feel sleepy or go to sleep during your procedure. Were you given anesthesia? Q11-OAS: Did your doctor or anyone from the facility explain the process of giving anesthesia in a way that was easy to understand? Q12-OAS: Did your doctor or anyone from the facility explain the possible side effects of the anesthesia in a way that was easy to understand? Q20-S: Did this anesthesiologist encourage you to ask questions? Q22-S: Did this anesthesiologist answer your questions in a way that was easy to understand? Q24-S: Did talking with this anesthesiologist during this visit make you feel more calm and relaxed? Q25-S: Using any number from 0 to 10, where 0 is the worst anesthesiologist possible and 10 is the best anesthesiologist possible, what number would you use to rate all your care from this anesthesiologist? |
“The department was the best I've seen as was the anesthesia.” | N/A |
| Staff interactions (including nurse) |
Q5-OAS: Were the clerks and receptionists at the facility as helpful as you thought they should be? Q6-OAS: Did the clerks and receptionists at the facility treat you with courtesy and respect? Q37-S: During these visits, did clerks and receptionists at this surgeon's office treat you with courtesy and respect? |
“…from start to finish the staff at this office was courteous, helpful and understanding.” “The staff was so accommodating and warm and I felt at ease the day of my procedure.” |
“Please BELIEVE the reviews. Never encountered such a rude, curt, unprofessional, and robotic front-line people-wouldn't dignify calling them team or staff.” “Staff was rude and unprofessional” |
| Physician interactions |
Q7-OAS: Did the doctors and nurses treat you with courtesy and respect? Q8-OAS: Did the doctors and nurses make sure you were as comfortable as possible? Q9-OAS: During your office visits before your surgery, did this surgeon listen carefully to you? Q11-S: During your office visits before your surgery, did this surgeon encourage you to ask questions? Q12-S: During your office visits before your surgery, did this surgeon show respect for what you had to say? Q15-S: After you arrived at the hospital or surgical facility, did this surgeon visit you before your surgery? Q17-S: Before you left the hospital or surgical facility, did this surgeon discuss the outcome of your surgery with you? Q31-S: After your surgery, did this surgeon listen carefully to you? Q33-S: After your surgery, did this surgeon encourage you to ask questions? Q34-S: After your surgery, did this surgeon show respect for what you had to say? |
“He was very courteous, personable, and very respectful of my needs.” “Both Dr. X and staff are very caring and professional and respect the patient's time and concerns.” “I felt that throughout the entire experience I was treated with the utmost respect. My voice was heard! I was treated with care and concern every step of the way!” “I feel like Dr. X really cares about you personally. She took the time to listen to my needs and was present throughout the entire process.” |
“… [the doctor] never even bothered to ask me if I have any other questions or concerns. There was also a total lack of communication during the procedure: the doctor simply did his routine without even warning me what I should expect. Such communication is essential when somebody is poking inside your ear with a sharp stick. When I felt uncomfortable after the procedure was done, he simply dismissed that and said it was normal.” “…he was brisk and abrupt and made me full uncomfortable about wasting his time.” “Untrustworthy and does unnecessary surgeries.” |
| Efficiency/time |
Q10-S: During your office visits before your surgery, did this surgeon spend enough time with you? Q3-S: After your surgery, did this surgeon spend enough time with you? |
“In this world of hurried, busy doctors, Dr. X takes time and truly listens to my concerns and offers an abundance of solutions.” | “The wait time is 2+ hours. Clearly they don't value your time.” “He rushed in/out, and made me feel like I was a hassle not a PAYING patient.” |
| Discharge instructions |
Q13-OAS: Discharge instructions include things like symptoms you should watch for after your procedure, instructions about your medicine, and home care. Before you left the facility, did you get written discharge instructions? Q14-OAS: Did your doctor or anyone from the facility prepare you for what to expect during your recovery? Q26-S: Did anyone in this surgeon's office explain what to expect during your recovery period? Q27-S: Did anyone in this surgeon's office warn you about any signs or symptoms that would need immediate medical attention during your recovery period? Q28-S: Did anyone in this surgeon's office give you easy to understand instructions about what to do during your recovery period? Q28-S: Did anyone in this surgeon's office give you easy to understand instructions about what to do during your recovery period? |
“He explained my surgery procedure and recovery in detail.” “The staff are so detailed and are home care packet has all of the information you would every want. I am 5 days out, and there have been no surprises as she explained everything to expect so thoroughly.” “We knew exactly what to expect once we got home and the doctor called to make sure everything was going all right that night!” |
“No aftercare instructions were given and these were requested for over a week. Finally, we just stopped requesting. I'm not sure even exactly what my son had done.” “When I asked how his ear looked he replied ‘infected.’ That was it…I had to stop him from leaving & ask if there was anything I needed to watch out for before our follow up in a week. Anything that would warrant a call or trip to ER. His reply ‘seizures, coma’ and left the room.” |
| Pain |
Q15-OAS: Some ways to control pain include prescription medicine, over-the-counter pain relievers or ice packs. Did your doctor or anyone from the facility give you information about what to do if you had pain as a result of your procedure? Q16-OAS: At any time after leaving the facility, did you have pain as a result of your procedure? |
“I had zero pain.” “She told me a head of time how much pain to expect and what I could do to help minimize it. However, when I got home, it ended up not being that bad! It was nice to know what to expect.” |
“…the pain was intense.” “…when he was finally ready to perform the procedure he was rather surly. It was very rough (a piece of my tongue was cut for a biopsy) and he didn't offer a prescription for pain meds. I don't consider myself a sissy but I was miserable for days.” |
| Nausea/vomiting |
Q17-OAS: Before you left the facility, did your doctor or anyone from the facility give you information about what to do if you had nausea or vomiting? Q18-OAS: At any time after leaving the facility, did you have nausea or vomiting as a result of either your procedure or the anesthesia? |
N/A | N/A |
| Bleeding |
Q19-OAS: Before you left the facility, did your doctor or anyone from the facility give you information about what to do if you had bleeding as a result of your procedure? Q20-OAS: At any time after leaving the facility, did you have bleeding as a result of your procedure? |
“..postop I had absolutely no pain or bleeding.” “They let us know I should call if it started bleeding overnight, but it was fine!” |
“I had a hard time with the bleeding.” “I saw him twice, and both times left with a bloody nose. I asked him to be gentle, but he said I must be just sensitive.” |
| Infection |
Q21-OAS: Possible signs of infection include fever, swelling, heat, drainage or redness. Before you left the facility, did your doctor or anyone from the facility give you information about what to do if you had possible signs of infection? Q22-OAS: At any time after leaving the facility, did you have any signs of infection? |
N/A | “I have gone through an infection so bad I was sent to the ER to get an IV.” |
| Overall |
Q23-OAS: Using any number from 0 to 10, where 0 is the worst facility possible and 10 is the best facility possible, what number would you use to rate this facility? Q35-S: Using any number from 0 to 10, where 0 is the worst surgeon possible and 10 is the best surgeon possible, what number would you use to rate all your care from this surgeon |
“I have a lot of trust in this surgeon and facility.” “I can't imagine a better experience with Dr. X, it was perfect and easy. I'd give 100 out of 5 stars if I could.” |
“I had a very bad experience.” “This guy gives me the creeps. Stay far away.” “If I could give negative stars, I would. Do your homework.” |
| Recommendation likelihood | Q24-OAS: Would you recommend this facility to your friends and family? | “A+ doctor, facility, and staff. I 100% recommend!” “My entire family has gone to Dr. X. You should too!” |
“Please run as fast as you can from this guy.” “Warning! Do not go here. I guarantee the people giving positive reviews have been offered a discount for a good review, because that's the only way this guy can get business.” |
CAHPS, Consumer Assessment of Healthcare Providers and Systems; N/A, not available; OAS-CAHPS, outpatient and ambulatory surgery CAHPS; S-CAHPS, surgical care CAHPS.
Appendix Table A8. Linking author-generated inductive themes to representative quotes from Healthgrades narrative review comments (n = 957)
| Theme | Positive comment example | Negative comment example |
|---|---|---|
| Bedside manner | “He has a warm, friendly, inviting bedside manner.” “[Doctor] is hands down equally as amazing, talented, caring and patient. She goes above and beyond to ensure all patient needs are met.” “He is amazing, super nice, confident in his work. He truly cares about his patients and wants to make sure their care/surgeries are successful.” “As a physician, he is kind and caring, always available, and gets to know each patient personally.” |
“Very dishonest and rude.” “…he was rude, snippy, and snappy at me. He made disdainful faces like a bratty little boy at me for no reason. He did all of the talking, & didn't ask me any questions about the my particular situation for which I was seeking treatment.” “He has a terrible bedside manner and is downright mean. Run as fast as you can. I wish I had!!!” He did all the talking. Didn't do an exam. And |
| Outcome (overall) | “I am ecstatic with my results.” “The results are life changing, thank you so much!” “At first, I was a little put off by Dr. X's bedside manner, but the results speak for themselves. I'm so happy!” “I couldn't have imagined a better outcome. I should have done this a decade ago. You are in good hands with Dr. X.” “I love it! She clearly did what we discussed, it's perfect.” |
“Did what HE wanted in surgery, I am very dissatisfied” “Better surgeons at your local butcher shop.” “When evaluating a physician, it doesn't really matter how ‘nice’ the doctor or staff is - the bottom line is the skill level and outcome. Learn from me. Don't be fooled, go elsewhere.” “More selling than results.” |
| Cosmetic outcome | “Gave me the cutest little nose, it's just what I wanted.” “My results were amazing. Dr. X took years off of my face and neck, all with no visible scarring.” “I was so nervous about the surgery, but now I am so glad I had it done. I am so happy with the results. It looks so natural, and I finally feel confident and great about how I look in pictures.” “My results were amazing. Dr. X took years off of my face and neck, all with no visible scarring…He performs customized procedures for the best outcome.” “…[Doctor X] has a wonderful gift of making me feel fresh and beautiful without looking too overdone.” |
“I live in horror everyday with the way I look since the surgery. I feel surgeries are taken very lightly without considering the traumatic affect they can have on the patient later on.” “If I could leave no stars I would. I'd also attach pictures of my very visible, lopsided and uneven nose.” “Botched' is the only word that can be used to clarify what I am living with after facial surgery which is having to be repaired by a more skilled surgeon.” “Because of how it turned out, even 6 months later I'm embarrassed to go outside…. It's making me extremely depressed and emotionally distressed.” |
| Functional outcome | “I went for a functional rhinoplasty that I had put off for way too long. Not only are the results, I believe are truly life changing.” “I've never breathed or slept this well in my life. I had no idea how sleep deprived I actually was! Since the surgery my snoring is gone, I'm dreaming, thinking clearer, and lost weight.” |
“Months later I'm still not able to breathe well. It's a scam.” “I ended up having nerve damage from the procedure.” “Although Dr. X did what he said he would, I believe he or his office should strongly warn future patients that there is a good chance they will lose all smell and taste for ever. I NEVER even thought this could happen… It is devastating.” |
| Answered questions | “She very much cares about her patients as individuals, making herself very available for questions in the weeks/months after my procedure.” “He really listened to what I wanted to accomplish and answered all of my questions. Dr. X explained which type of surgery could help achieve my desired goals.” |
“He did not explain my condition and he was annoyed when I asked questions to try to understand.” “The consultation was a joke. He pressured me several times to go on Instagram to talk live about my procedure. (I did not consent to this.) How could I ask/answer personal questions while ‘live’ on social media.” |
| Follow-up | “They are very caring and thorough, and have given me excellent follow up care.” “I just came back from my 3-month follow-up appointment, and it was wonderful! Dr. X and his staff was so excited to see how while I'm doing. It feels like they are family and really care!” |
“I was not encouraged to return or follow up” “He basically dropped me after the procedure- gave me no education about the condition, and I was not encouraged to return or ever follow up.” “… I cannot even get office to call me back…they collected money, and abandoned me.” |
| Knowledge | “She is very knowledgeable about current studies in the field and shared some of these with me” “From day one her knowledge and thoroughness impressed me. She spends so much time reviewing what I would like to see as the end result and setting proper expectations. Sign of a smart doctor.” “She is an expert in the field, and knows the latest techniques. She was trained at a top hospital, so know you are getting the best care and the real deal.” |
“He has no knowledge about the field in which he practices. It's scary.” “She did not spend the necessary time with me to evaluate and just looked at me from across her desk. She would not admit she was over her head and just recommended someone else.” “Do not be taken in by the smooth presentation or persona of this ‘charming’ man as an it is entirely unrelated to his level of expertise or surgical skills.” |
| Finances | “My surgery was worth every penny.” “I loved how they are transparent about cost up front and help you budget ahead. This is so helpful so you know what to plan for and expect. Very few surgeons do this.” “When you are in her office she makes you feel like her top priority. She is never distracted. You are getting what you pay for.” |
“This Dr. cares about one thing only–money. Patients and their wellbeing mean nothing. This businessman will misdiagnose you, provide false information and data, and then overbill you.” “Stay from this money hungry businessman.” “…we have gone through a fair amount of financial stress due to billing issues.” |
| Personal communication | “He was right there after when I had an issue, twice over the weekend and very accessible after the fact in his office and on the phone or even an email” “Dr. X gave her personal cell phone to call with any urgent matter even though she was on vacation!” |
“I called his office for help, just days after the surgery and I couldn't get anyone to answer. I ended up going to the hospital instead.” “When I got home on a Friday I realized we didn't even have a number to call if a problem came up (which it did).” |
Authors' Contributions
The authors confirm that all listed authors had substantial contributions to the conception or design of the study; or the acquisition, analysis, or interpretation of data for the study; and drafting the article or revising it critically for important intellectual content; and final approval of the version to be published; and agree to be accountable for all aspects of the study in ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.
Author Disclosure Statement
The authors confirm that they do not have competing interests, financial interests, funding, employment, or other competing interests that may affect the integrity of the research reported.
Funding Information
K.A.M. is supported by the National Institute of Health Agency for Healthcare Research and Quality under award no. T32HS026116-02. There were no other funding sources for any authors.
Given that facial plastic surgeons have a high proportion of self-pay patients, these patient experience surveys are not routinely used for most facial plastic procedures.
References
- 1. Terlutter R, Bidmon S, Röttl J. Who uses physician-rating websites? Differences in sociodemographic variables, psychographic variables, and health status of users and nonusers of physician-rating websites. J Med Internet Res. 2014;16(3):e97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014;311(7):734–735 [DOI] [PubMed] [Google Scholar]
- 3. Sobin L, Goyal P. Trends of online ratings of otolaryngologists: what do your patients really think of you? JAMA Otolaryngol Head Neck Surg. 2014;140(7):635–638 [DOI] [PubMed] [Google Scholar]
- 4. Woodward C. “Anti-defamation” group seeks to tame the rambunctious world of online doctor reviews. CMAJ. 2009;180(10):1010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Paavola A. Miami plastic surgeon sues former patients for bad Yelp reviews. Becker's Hospital Review. https://www.beckershospitalreview.com/legal-regulatory-issues/miami-plastic-surgeon-sues-former-patients-for-bad-yelp-reviews.html. Accessed September1, 2019
- 6. Kadry B, Chu LF, Kadry B, Gammas D, Macario A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011;13(4):e95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Lewis P, Kobayashi E, Gupta S. An online review of plastic surgeons in southern California. Ann Plast Surg. 2015;74(Suppl. 1):S66–S70 [DOI] [PubMed] [Google Scholar]
- 8. CAHPS Patient Experience Surveys and Guidance. Rockville, MD: Agency for Healthcare Research and Quality. https://www.ahrq.gov/cahps/surveys-guidance/index.html. Accessed September1, 2019
- 9. Press Ganey® Associates Medical Practice. www.pressganey.com/index.aspx. Accessed September4, 2019
- 10. Quality Payment Program. US Centers for Medicare and Medicaid Services. www.qpp.cms.gov. Accessed September1, 2019
- 11. Physician Compare. US Centers for Medicare and Medicaid Services. https://www.medicare.gov/physiciancompare/#. Accessed September1, 2019
- 12. Ranard BL, Werner RM, Antanavicius T, et al. Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Aff (Millwood). 2016;35(4):697–705 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Best Practices for Mixed Methods Research in the Health Sciences. Bethesda (Maryland): National Institutes of Health; 2013: 541–545 [Google Scholar]
- 14. American Academy of Facial Plastic and Reconstructive Surgery. American Academy of Facial Plastic and Reconstructive Surgery, Inc. www.aafprs.org. Accessed May1, 2019
- 15. Patient Satisfaction. Healthgrades Operating Company, Inc. https://www.healthgrades.com/quality/patient-satisfaction. Accessed May1, 2019
- 16. About Zocdoc. Zocdoc, Inc. http://support.zocdoc.com. Accessed May1, 2019
- 17. Google My Business Help: Score ratings for local places. Google LLC. https://support.google.com/business/answer/4801187?hl=en. Accessed May1, 2019
- 18. Vitals.com: Find a Doctor, Doctor Reviews & Ratings. Vitals Consumer Services, LLC. www.vitals.com. Accessed May1, 2019
- 19. Provider Profile. Healthgrades Operating Company, Inc. https://helpcenter.healthgrades.com/help/provider-profile. Accessed May1, 2019
- 20. Geography Program. United States Census Bureau. https://www.census.gov/geo/reference/gtc/gtc_census_divreg.html?cssp=SERP. Accessed September1, 2019
- 21. Bardach NS, Asteria-Peñaloza R, Boscardin WJ, Dudley RA. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Qual Saf. 2013;22(3):194–202 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Shemirani NL, Castrillon J. Negative and positive online patient review of physician- 1 vs 5 stars. JAMA Facial Plast Surg. 2017;19(5):435–436 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Boss EF, Thompson RE. Patient satisfaction in otolaryngology: can academic institutions compete? Laryngoscope. 2012;122(5):1000–1009 [DOI] [PubMed] [Google Scholar]
- 24. Dorfman RG, Purnell C, Qiu C, Ellis MF, Basu CB, Kim JYS. Happy and unhappy patients: a quantitative analysis of online plastic surgeon reviews for breast augmentation. Plast Reconstr Surg. 2018;141(5):663e–673e [DOI] [PubMed] [Google Scholar]
- 25. Agarwal AK, Pulullo AP, Merchant RM. “Told”: the word most correlated to negative online hospital reviews. J Gen Intern Med. 2019;34(7):1079–1080 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Malbon J. Taking fake online consumer reviews seriously. J Consum Policy. 2013;36(2):139–157 [Google Scholar]
- 27. Bialer J. What I learned from scraping every single ZocDoc doctor. NY Data Science Academy. https://nycdatascience.com/blog/student-works/web-scraping/analyzing-zoc-doc-doctors/. Accessed September1, 2019
- 28. Ryan T, Specht J, Smith S, DelGaudio JM. Does the press ganey survey correlate to online health grades for a major academic otolaryngology department? Otolaryngol Head Neck Surg. 2016;155(3):411–415 [DOI] [PubMed] [Google Scholar]
- 29. Vaismoradi M, Turunen H, Bondas T. Content analysis and thematic analysis: implications for conducting a qualitative descriptive study. Nurs Health Sci. 2013;15(3):398–405 [DOI] [PubMed] [Google Scholar]