Online Ratings and Perceptions of Pediatric Otolaryngologists

Abstract

Objective:

To assess and characterize online ratings and comments of pediatric otolaryngologists and determine factors that correlate with higher ratings.

Study Design:

Online database analysis.

Methods:

All American Society of Pediatric Otolaryngology (ASPO) members were queried on Healthgrades, Vitals, RateMDs, and Yelp for their online ratings and comments as of June 2020. Ratings were normalized for comparison on a five-point Likert scale. All comments were categorized based on context and positive or negative quality.

Results:

Of the 561 ASPO members, 489 (87%) were rated on at least one online platform. Of those rated, 410 (84%) were on Healthgrades, 429 (88%) on Vitals, 236 (48%) on RateMDs, and 72 (15%) on Yelp. Across all platforms, the average overall rating was 4.13 ± 0.03 (range: 1.00–5.00). We found significant positive correlations between overall ratings and specific ratings (P< 0.001) on all individual topics. In addition, the majority of all narrative comments were related to perceived physician bedside manner and clinical outcome, with negative comments correlating negatively with overall score (P< 0.05). Time spent with the physician was the only category in which both positive and negative comments showed significant correlation with the overall physician rating (P = 0.016 and P = 0.017, respectively). Attending a top-ranked medical school or residency program did not correlate with higher or lower ratings.

Conclusion:

Online ratings and comments for pediatric otolaryngologists are largely influenced by patient and parent perceptions of physician competence, comforting bedside manner, and office and time management.

Introduction

Physician rating websites have increasingly become commonly utilized tools for patients to publicly provide feedback on physician performance and to obtain information about potential health care providers. According to a study released by the Pew Internet Project and California HealthCare Foundation in 2010, approximately 80% of Americans access health-related information on the Internet, with 44% of Internet users searching for information related to health care professionals.¹ An increasing number of patients have been shown to view online physician rating websites prior to their initial visit, with a 2014 study reporting 35% selecting a physician based on “good” ratings, and 37% avoiding a physician with “bad” ratings.^2–5 Despite some skepticism regarding their correlation with validated measures of patient satisfaction,^6–8 physician rating websites nonetheless have the potential to influence patient preferences and encourage improved care among health care providers.^4,5,9

Online physician ratings and comments have previously been explored and characterized generally in the literature. Prior studies have found that almost 90% of physician ratings/reviews are positive while 6% are negative.^10,11 Similarly, other reports have shown that anywhere from 63 to 89% of narrative comments are positive.^12,13 However, it should be noted that only a small percentage of physicians had been rated on a physician rating website at the time of these reports in 2010 and 2012.^10–14 More recently, online physician ratings have been investigated by specialty in various fields, including in orthopedic surgery,^15,16 plastic surgery,^17,18 pediatrics,¹⁹ and otolaryngology.^9,20 To our knowledge, most of the current literature that examines online ratings reports adult patients’ use of online physician websites for their personal care. One study by Hanauer et al.¹⁹ examined parental awareness and utilization of physician rating sites through a cross-sectional survey. The authors concluded that parents are significantly more likely to select a neighbor’s “recommended” physician when exposed to positive online physician ratings, and are less likely to choose the neighbor’s recommendation when subjected to viewing negative online ratings.¹⁹

To date, no study has investigated online ratings or comments exclusively for pediatric otolaryngologists. We aimed to assess and characterize patients’ online ratings and comments of pediatric otolaryngologists and determine possible predictive factors for higher overall physician ratings. Analysis of online physician website ratings may be useful for the patients, parents, and physicians, revealing the most valuable aspects of patient care and treatment outcomes from the patient’s or parent’s perspective.

Materials and Methods

Following Institutional Review Board approval from the University of California Irvine, all American Society of Pediatric Otolaryngology (ASPO) members were queried for ratings and comments on Healthgrades, Vitals, RateMDs, and Yelp made prior to and until June 2020. Each physician’s sex, years in practice (or years of experience), state of practice, medical school and residency program attended, and rating criteria provided by the websites were extracted for analysis. Ratings for each individual ASPO member were collected directly from the websites and normalized for comparison on a five-point Likert scale, while a weighted overall rating for each pediatric otolaryngologist was additionally calculated via the following formula: [(Healthgrades rating × Number of Healthgrades votes) + (Vitals rating × Number of Vitals votes) + (RateMDs rating × Number of RateMDs votes) + (Yelp rating × Number of Yelp votes)]/(Total number of votes across the four platforms). All comments were categorized by context and by positive or negative quality. States of practice were grouped into four geographical regions: Northeast, Midwest, South, and West. The 2019 U.S. News and World Report Rankings were used to rank the physicians’ medical schools and residency programs. Statistical analysis was performed using PASW Statistics 25.0 software (SPSS Inc., Chicago, IL) with P < 0.05 considered statistically significant. Pearson correlation, Spearman correlation, independent samples t-test, and one-way analysis of variance (ANOVA) were used to compare continuous and/or ordinal variables.

Results

Of the 561 ASPO members, 489 (87%) were rated on at least one online platform. Of those with ratings, 410 (84%) were on Healthgrades, 429 (88%) on Vitals, 236 (48%) on RateMDs, and 72 (15%) on Yelp. Across all platforms, the average overall rating was 4.13 ± 0.03 (range: 1.00–5.00) (Table 1). The average number of ratings per pediatric otolaryngologist was 23.24 ± 1.30 (median = 16.0), while the average number of comments per rated otolaryngologist was 9.60 ± 0.72 (median = 5.0) (Table 1). The 25th and 75th percentiles for overall rating were 3.68 and 4.65, respectively. There were statistically significant positive correlations between overall rating and ratings on all of the individual subcategories (P< 0.001), including physician trustworthiness/friendliness, knowledge answering questions, adequate time spent with the patient, and office staff and scheduling efficiency (Table 2). Overall ratings according to the physicians’ geographic region of practice were as follows, with number and percentage of physicians in parentheses: Northeast (109, 22.3%): 4.13 ± 0.07, Midwest (108, 22.1%): 4.14 ± 0.07, South (176, 36.0%): 4.16 ± 0.05, and West (75, 15.3%): 3.96 ± 0.10. One-way ANOVA showed no significant difference in overall rating and region of practice (P = 0.236).

Table 1.

Online ratings of the American Society of Pediatric Otolaryngology members across various rating platforms

Rating Website	No. of Rated Pediatric Otolaryngologists	Average Overall Rating Score (SD)	Average No. of Raters per Pediatric Otolaryngologist with Rating (SD)	Average No. of Comments per Rated Pediatric Otolaryngologist (SD)
Healthgrades	410 (73.1%)	4.08 (0.89)	8.72 (9.08)	2.44 (4.09)
Vitals	429 (76.5%)	4.19 (0.76)	13.96 (20.29)	4.44 (9.54)
RateMDs	236 (42.1%)	4.11 (1.46)	6.00 (9.86)	6.00 (9.86)
Yelp	72 (12.8%)	3.49 (1.48)	5.20 (8.18)	5.20 (8.18)

SD: standard deviation.

Table 2.

Association between specific factors underlying overall pediatric otolaryngologist ratings and the weighted overall rating

Criteria	Average rating (SD)	P value	r *
Healthgrades
Physician’s trustworthiness	4.18 (0.87)	<0.001	0.761
Explaining conditions well	4.18 (0.87)	<0.001	0.751
Answering questions	4.15 (0.90)	<0.001	0.756
Time well spent	4.15 (0.89)	<0.001	0.756
Office scheduling	4.22 (2.60)	<0.001	0.196
Office environment	4.34 (0.69)	<0.001	0.552
Staff friendliness	4.27 (0.74)	<0.001	0.569
Vitals
Quality rating	4.63 (0.66)	<0.001	0.696
Easy appointment	4.22 (0.81)	<0.001	0.531
Promptness	4.29 (2.33)	<0.001	0.183
Friendliness	4.50 (2.64)	<0.001	0.147
Accurate diagnosis	4.38 (0.84)	<0.001	0.638
Bedside manner	4.25 (0.97)	<0.001	0.661
Spending adequate time	4.24 (0.93)	<0.001	0.676
Appropriate follow-up	4.18 (0.92)	<0.001	0.671
Wait-time in minutes	19.47 (11.09)	<0.001	−0.302
RateMDs
Staff	4.01 (1.14)	<0.001	0.435
Punctuality	3.89 (1.08)	<0.001	0.422
Helpfulness	3.96 (1.21)	<0.001	0.554
Knowledge	4.15 (1.11)	<0.001	0.549
Yelp	3.49 (1.48)	<0.001	0.526

^*Pearson’s bivariate correlation was performed comparing each specific criterion with the weighted overall rating.

SD: standard deviation.

Conversely, Spearman rank correlation demonstrated a significant negative correlation between years of experience (mean: 26.59 ± 0.52, range: 4–63) and weighted overall rating (Spearman’s P = 0.001, R = −0.159). Total number of votes per provider also negatively correlated with weighted overall rating (Spearman’s P = 0.001, R= −0.224). Independent samples t-test, however, demonstrated that pediatric otolaryngologists who attended a top-50 medical school (both research [P = 0.955] and primary care [P = 0.483]) or residency [P = 0.262]) did not differ in overall ratings compared with those who did not attend a top-50 program. Comparable results were obtained when broken down by top-25 programs (Table 3).

Table 3.

Association between attendance at top-ranked medical school and residency programs and physicians’ weighted overall rating

Criteria	Mean Score of Those Meeting Criteria (n)	Mean Score of Those Not Meeting Criteria (n)	P value*
Top-50 medical school (research)	4.13 (248)	4.12 (241)	0.955
Top-50 medical school (primary care)	4.15 (237)	4.10 (252)	0.483
Top-50 otolaryngology residency program	4.17 (224)	4.09 (265)	0.262
Top-25 medical school (research)	4.15 (141)	4.12 (348)	0.610
Top-25 medical school (primary care)	4.11 (127)	4.13 (362)	0.770
Top-25 otolaryngology residency program	4.16 (134)	4.11 (355)	0.561

^*Results were calculated via independent sample t-test.

A total of 3,504 narrative comments across four platforms (1,001 on Healthgrades; 1,898 on Vitals; 236 on RateMDs; and 369 on Yelp) were analyzed and categorized (Table 4). Since the categories were not mutually exclusive, (e.g., one comment could meet the criteria of multiple categories) this yielded a total of 9,287 category entries. Of these entries, 6,590 (71.0%) and 2,697 (29.0%) were characterized as positive and negative comments, respectively.

Table 4.

Comment categorization and the respective number of comments containing the underlying theme (not mutually exclusive)

	Positive Comments		Negative Comments
Comment category	n (Mean ± SD per Physician)	P value (r)a	n (Mean ± SD per Physician)	P value (r)a
Professionalism, communication, answering questions	1198 (3.89 ± 6.25)	0.122 (0.067)	292 (2.25 ± 1.95)	0.012 (−0.199)*
Clinical outcome	1839 (5.27 ± 9.61)	0.258 (0.035)	523 (2.59 ± 2.95)	0.011 (−0.162)*
Friendliness, caring, and feeling comfortable	2131 (5.52 ± 7.24)	0.063 (0.078)	636 (3.48 ± 3.91)	<0.001 (−0.357)*
Spending time	523 (2.53 ± 2.35)	0.016 (0.149)*	295 (2.30 ± 3.01)	0.017 (−0.188)*
Wait time	147 (1.81 ± 2.44)	0.200 (−0.095)	315 (2.28 ± 2.73)	0.330 (−0.038)
Helpful and friendly staff	642 (3.12 ± 5.71)	0.299 (0.037)	338 (2.86 ± 4.21)	0.055 (−0.148)
Getting appointments and follow-ups	95 (1.51 ± 0.98)	0.253 (0.085)	163 (1.87 ± 1.98)	0.256 (−0.071)
Cost and insurance difficulties	15 (1.67 ± 1.00)	0.348 (−0.152)	135 (2.18 ± 2.57)	0.338(−0.054)

^aPearson’s bivariate correlation was performed to show the effect of comment categories on the physician’s weighted overall rating.

SD: standard deviation.

Discussion

The majority of ASPO members (87%) have at least one online review, with most physicians having ratings on Healthgrades (73.1%) or Vitals (76.5%). Overall ratings are generally high with an average of 4.13 ± 0.03 out of 5. These findings are largely consistent with the results reported by Sobin and Goyal⁹ and Goshtasbi et al.²⁰ Sobin and Goyal⁹ queried 281 academic otolaryngologists in the Northeastern United States in 2013 and found that 186 (69.9%) physicians rated on Healthgrades and 202 (81.8%) rated on Vitals had average ratings of 4.4 and 4.25, respectively. Goshtasbi et al.²⁰ examined all 560 members of the American Neurotology Society (ANS) in 2018 and found that there were 420 (75.0%) rated on Healthgrades and 392 (70.0%) rated on Vitals, with average ratings of 3.96 and 4.05, respectively. The higher average ratings of Sobin and Goyal’s cohort as compared with those of Goshtasbi et al. and our cohort (4.08 and 4.19 for Healthgrades and Vitals, respectively) could be attributed to the greater than five-year disparity in data collection and subsequent inclusion of more reviews.

It was previously shown that there were an average of 3.68 and 4.30 ratings per pediatric otolaryngologist on Healthgrades and Vitals, respectively.⁹ This is in contrast to our finding that there was an average of 8.72 and 13.96 ratings per practitioner profile on Healthgrades and Vitals, respectively. Alternatively, the difference in average ratings amongst studies may also be the result of variances in physician inclusion criteria. Goshtasbi et al.²⁰ queried all ANS members while we similarly queried all ASPO members, regardless of professional affiliations, whereas Sobin and Goyal only identified academic faculty members who were part of a small subset of otolaryngology programs in the Northeast US region.

Medical school ranking and residency ranking categorized by top-25 or top-50 institution did not influence ratings or comments, a finding consistent with other studies.^9,20 State of residence and practice also did not correlate with average rating, similar to findings by Sobin and Goyal.⁹ In contrast to prior studies; however, we found that years in practice negatively correlated with overall rating. This finding is likely a result of physicians who have been in practice longer having a larger pool of patients and therefore a higher total of negative ratings and online comments. We can also speculate that pediatric otolaryngologists who are early in their careers are more likely to spend an “adequate” amount of time with patients compared to more experienced otolaryngologists, a comment category found to be positively correlated with overall physician rating.

This notion that years of experience may influence the performance of physicians has previously been hypothesized in the literature.^21,22 Similar to our findings, Choudhry et al.²¹ showed an inverse relationship between the number of years a physician has been in practice and the quality of care provided. In a systematic review that included 59 studies, Choudhry et al.²¹ determined that providers who were older or had more years of experience seemed to follow current standards of care less closely, resulting in lower performance outcomes than younger or less experienced physicians. In a separate study, however, Schiff et al.²² found that pediatricians who have been practicing longer are more comfortable recognizing and diagnosing dysphonia and are also more likely to refer dysphonic patients to pediatric otolaryngologists. While these findings suggest that more experienced pediatricians provide an increased quality of care to their patients, it is important to note that the study was conducted using surveys completed by the physicians and did not include treatment outcome measures.

We found statistically significant correlations between overall average rating and ratings on all of the individual subcategories. The factors with the strongest correlation to overall rating (r ≥ 0.6, P < 0.001) were the physician’s trustworthiness, ability to explain and answer questions, accurate diagnosis, bedside manner, adequate time spent with the patient and family, and appropriate follow-up. In general, physician knowledge leading to an accurate diagnosis, bedside manner, and adequate time spent with the patient, are common factors among multiple studies that have shown to be the most notable predictors of higher ratings.^15,20 This demonstrates that aspects related to the physician-patient-family interaction play just as significant a role as physician knowledge in patients’ and families’ online rated satisfaction.

Among the 3,504 narrative comments evaluated online, we found that the highest number of both positive and negative comments left by raters across all platforms was related to bedside manner and clinical outcome. This result has previously been demonstrated in studies from 2018²⁰ and 2012.²³ Emmert et al. evaluated 3,000 randomly selected narrative comments from the German physician rating website Jameda and determined that professional competence of the physician made up the most frequently mentioned concern (62.5%).²³ Friendliness and caring attitude of the physician was the second most frequently mentioned concern (38.3%) amongst the positive and negative comments.²³

Although the number of positive comments online greatly outnumbered the negative ones, we found that more negative comment categories on the pediatric otolaryngologists’ profiles showed statistically significant correlations with the physicians’ overall rating. A negative correlation was observed between the pediatric otolaryngologists’ overall score and the number of negative comments concerning perceived professionalism, communication, clinical outcomes, bedside manner, and spending time with the patient/family. A majority of the categorized positive comments did not provide any statistically significant correlation in our study, suggesting that negative comments across all platforms had the greatest effect on the pediatric otolaryngologists’ overall score. However, both positive and negative comments related to time spent with the physician showed a significant correlation with the overall physician rating (P = 0.016 and P = 0.017, respectively), suggesting that parents value an adequate amount of time spent with the physician over positive experiences related to perceived physician competence and office/insurance interactions.

The greatest number of negative comments were related to physician friendliness and caring manner (23.6%) and clinical outcome (19.4%), indicating once again that the negative ratings that most influence overall scores are related to bedside manner more so than the quality of care and clinical outcome. Comments related to superior or substandard office management, such as those regarding office wait time, staff friendliness/helpfulness, ease of appointment and follow-ups, and cost or insurance complaints, did not significantly contribute to the overall physician rating, unlike in prior studies in which the patient was likely to also be the online rater.²⁰ This suggests that while adult patients may care more about office staff interactions and wait times, these factors are not necessarily as important to the caregivers of pediatric patients, who are presumably the online raters of pediatric otolaryngologists. Parents are seemingly more concerned about physician bedside manner and time spent with the physician rather than external sources of patient satisfaction or dissatisfaction (e.g., interactions with office staff, appointment scheduling, insurance difficulties).

One of the main limitations to online platforms is their composition of largely subjective views which may not necessarily correlate with objective measures. Patient website reporting is also likely to come from a biased selection of patients or caregivers who choose to compose online reviews, and furthermore, patients who have had particularly positive or negative experiences may be more inclined to rate or comment on their physicians relative to other patients. Despite the biases and limitations, however, online physician ratings should be regarded as measures of patient experience and may be useful tools for patients when selecting healthcare providers and for clinicians to improve the quality of their care.^5,24

Conclusion

Online ratings and comments for pediatric otolaryngologists are highly dependent on patient and parent perceptions of physician competence, comforting bedside manner, and time spent with the physician. While these perceptions impact online ratings and the positive or negative quality of comments, we determined that minimizing the number of negative comments, especially regarding perceived physician bedside manner, clinical outcome, and time spent with the physician, and maximizing the number of positive comments related to time spent with the physician, leads to higher overall scores and online perception. Our study underscores the importance of recognizing the specific factors that impact the patient and parent experience, and those that contribute most to the overall online ratings for pediatric otolaryngologists.