Physician Rating Websites: an Analysis of Physician Evaluation and Physician Perception

Abstract

Background:

The goal of this study was to evaluate current physician ratings websites (PRWs) to determine which factors correlated to higher physician scores and evaluate physician perspective of PRWs.

Methods:

This study evaluated two popular websites, Healthgrades.com and Vitals.com, to gather information on practicing physician members of the American Shoulder and Elbow Society database. A survey was conducted of the American Shoulder and Elbow Society (ASES) membership to gather data on the perception held by individual physicians regarding PRWs.

Results:

We found that patients were more likely to give physicians positive reviews and the average overall score was 8.35 (3.75-10). Patient wait time (P=0.052) trended toward significance as a major factor in determining the overall scores, while ratings in both physician bedside manner (P=0.001) and physician/staff courtesy (P=0.002) were significant in reflecting the overall score given to the physician. According to our survey, a majority of the respondents were indifferent to highly unfavorable to PRWs (88%) and the validity of their ratings (78%).

Conclusion:

As PRWs become increasingly popular amongst patients in this digital age, it is critical to understand that the scores are not reflective of a significant proportion of the physicians’ patient population. Physicians can use this study to determine what affects a patient’s experience and focus efforts on improving patients’ perception of quality, overall satisfaction, and overall care. Consumers may use this study to increase their awareness of the potential for significant sampling error inherent in PRWs when making decisions about their care.

Introduction

As the accessibility and use of the internet grows; patients will continue to influence the perceived quality of the healthcare industry by using online resources as a platform, specifically physician-rating websites (PRWs). In 2010, 88% of Americans used the Internet to search for health related information, out of which, 47% looked up information about their physicians (1). Up to 65% of patients that used PRWs ultimately selected a physician for their medical care based on that rating (2).

The primary objectives of PRWs are to provide demographic information regarding the physicians and their practices. In addition they provide patient driven information regarding their overall experience with the physician and their office. The increasing popularity of these websites may seriously impact a physicians practice in a negative or positive manner. Disgruntled or frustrated patients may give undeserving bad reviews based on a single experience that could then discourage other patients from seeking care from that particular physician. Prior studies indicate that patients tend to submit an evaluation if they have a strong positive or strong negative experience, suggesting a bimodal distribution of the ratings (3). The ratings that physicians receive online may be very small relative to the overall patient population under that physician’s care. Studies have shown that a low number (7%) of patients actually report their experience online following an office visit (4). However, studies have also shown that most patient reviews are positively skewed (5). One study shows that 67% of reviews report a rating of 75 or higher, which is overwhelmingly positive (1).

A similar study was conducted in which hand surgeons were reviewed online. It showed that the average overall rating for physicians was 8.1 out of 10 across two different websites (Healthgrades.com and Vitals.com) (6).

While PRWs are an avenue for patients to evaluate their physicians, it is also an opportunity for physicians to market their clinical care. Currently, it is unclear whether physicians actually use these types of websites to promote and build their practice. The purpose of this study was to assess physician-rating websites based on the number of reviews and to determine factors correlating to the overall physician scores. Furthermore, we conducted a detailed survey of members of the American Shoulder and Elbow Society (ASES) to evaluate physician perception of PRWs. We hypothesized that there would be a minimal number of reviews (< 1% of new patients) relative to the overall patient population in a physician’s service area during a specified time period and that the reviewers would be more likely to report a strong positive or negative review. In addition we hypothesize that physicians would have an unfavorable perception of PRW’s.

Materials and Methods

This study evaluated two popular websites, Healthgrades.com and Vitals.com. We gathered information on practicing physicians from the American Shoulder and Elbow Society database (n=299, 288 men, 11 women). First and last names of physicians were entered into the search bars on each respective website to locate the categorical scores. The overall score, number of reviews, and wait times were reported for each physician. Additionally, scores for overlapping categories between the websites, such as ease of making appointments, courtesy of staff, and bedside manner/listens and answers questions were also recorded. Reviews on Healthgrades.com are reported on a 5-star scale, while reviews on Vitals.com are reported on a 4 star scale. Both of these evaluations were converted to a 10-point scale for data analysis and comparison. An analysis of the reviewer comments (only available for Vitals) was also conducted. Two reviewers (ACN and UMS) categorized the comments as positive, negative, or neutral. The data was analyzed using a mixed linear regression for all factors correlating with overall scores, number of reviews, and components of overall score. Box-Cox transformations (raised to 2.5 power) were utilized to allow the linear regression assumptions to be satisfied. Additionally, an intraclass correlation coefficient (ICC) was used to compare Healthgrade.com and Vitals.com.

A survey of the American Shoulder and Elbow Society (ASES) membership was conducted in order to evaluate physician perspective of PRWs. Survey questions included: years of experience, practice region, practice setting, impression and validity of PRWs, perceptions of the Affordable Care Act, and the benefits/concerns of PRWs. For qualitative answers dealing with individual concerns and benefits regarding PRWs, two reviewers (ACN and UMS) independently categorized the responses. A third reviewer (JAA) adjudicated any differences in categorization.

Results

PRW Analysis

We discovered that patients were more likely to give positive reviews with an average of 8.35 (3.75-10) thus refuting our hypothesis [Figure 1]. Only 2% of the physicians we reviewed had an average overall score of less than 5. Website (P=0.051) and gender (P=0.055) played a large part in determining the overall score, and this trended toward significance. Patient who reviewed their physician on Vitals.com were more likely to give higher scores than those who reviewed on Healthgrades.com. Male physicians were more likely to get higher scores than females. We found that geographical region, age, and number of reviews did not show a statistically significant impact on overall scores [Figure 2]. The agreement between Healthgrades.com and Vitals.com was moderate (ICC= 0.48).

Figure 1.

Overall Score Distribution

Figure 2.

Correlation of age, number of reviews, region, and gender with overall score

Healthgrades.com had an average of 20 reviews per physician (range: 1-114), while Vitals.com had an average of 18 reviews (range: 1- 76). Notably, 5% (15) of physicians could not be found on Healtgrades.com, while 13.7% (41) were not found on Vitals.com. Physicians in the Northeast regions had significantly more reviews than the Midwest, South, and West regions (P=.04). In addition older physicians were more likely to get fewer reviews compared to their younger counterparts (P=.033) [Figure 3]. Wait time (P=0.052) trended toward significance in determining overall scores, while ease of making appointment (P=0.119) did not significantly correlate with overall score, contrary to our hypothesis. However, both physician bedside manner (P<<0.05) and physician/staff courtesy (P=0.002) were significant in reflecting the overall score of the physician [Figure 4].

Figure 3.

Correlation of age and region with number of responses

Figure 4.

Correlation of individual categorical factors with overall score

ASES Survey Analysis

One hundred out of the current 447 ASES members (22%) responded to the questionnaire. Forty-six of the members that filled out the survey had greater than 20 years of experience, while the remaining had between 5-20 years of experience. The distribution of practice regions represented was similar (Northeast: 33%, Midwest: 19%, South: 26%, and West: 22%). The majority of physicians who filled out the survey were either from an orthopedic group or academic institution setting (78%).

Seventy-four percent of the physicians responded as not regularly checking their online PRW profile. A majority of the respondents were indifferent, unfavorable, or highly unfavorable to PRWs, the Affordable Care Act withholding funds for low satisfaction scores, and the validity of their ratings [Table 1]. Ninety-four percent of the respondents said they made no changes in their practice based on the reviews [Table 2]. We found that the 6% who made changes to their practice addressed interpersonal skills with their staff, reduced wait times, and worked on their communication strategies. Interestingly, one of the respondents admitted to having paid to remove a negative review on a PRW.

Table 1.

Physicians’ impressions of PRWs, Affordable Care Act, and validity of ratings

Question	Highly Favorable (%)	Favorable (%)	Indifferent (%)	Unfavorable (%)	Highly unfavorable (%)
What is your impression of physician rating websites?	2	10	48	27	13
The Affordable Care Act has implemented a policy to withhold funds for low patient satisfaction scores. What is your impression of this?	0	5	20	36	39
What is your impression on the validity of your ratings?	3	19	40	18	20

Table 2.

Physicians’ responses to various factor relating to PRWs

Question	Yes (%)	No (%)
Do you regularly check your online profile on physician rating websites?	26	74
Have you ever asked a patient to leave a positive review on a physician rating website?	23	77
Do you think online ratings have helped your practice?	37	63
Have you changed your practice based on your ratings?	6	94
Have you ever paid to remove a negative review?	1	99

While categorizing the qualitative responses, the majority of the concerns about PRWs dealt with “lack of validation” (47%), “low sample size” (25%), “negatively biased reviewers” (47%), “lack of avenue to respond” (13%), and “misrepresentation of their practice” (8%). The benefits were characterized into the following categories: “marketing” (65%), “providing relevant information” (28%), and “providing feedback” (10%).

Discussion

The internet has become a powerful tool for healthcare consumers in our current environment. Healthcare information is readily available for consumers for self-diagnosis and for seeking physician care (3). There is a lack of publicly available information and transparency on the quality of healthcare providers (2). This allows healthcare consumers an open platform to evaluate and rate the healthcare provider and services to aid each other in the physician selection process. The PRWs currently used by consumers allows patients to rate providers in an attempt to influence their care and inform others regarding their experience. The use of PRWs is expected to grow as healthcare technology progresses and becomes more accessible to healthcare consumers (3). PRWs have become an integral part of assessing and selecting a physician and could potentially add to the publicly available information on physician quality (2). However, due to low percentage of reviews, the physician ratings do not adequately represent the quality of physician care due to the small sample size and studies have concluded that they are not a meaningful scale for patients (2, 7).

Contrary to our hypothesis, the overall scores were generally positive. Although, it would typically be expected for ratings to have a bimodal distribution, studies have shown a positively skewed distribution (5). This could be due to the likelihood that patients with overwhelming positive experiences are more likely to evaluate their physicians than patients having a negative experience.

The average numbers of reviews for both PRWs was less than 20. This is in agreement with a prior study, reporting less than 2.4 reviews after examining the ratings of over 500 urologists (8). This substantiates physician concern about a low sample size present in the PRWs and is a cause for physician to have an unfavorable impression of PRWs and the validity of their ratings.

Physicians in the Northeast regions had significantly more reviews than all other regions. This could be due to multiple factors such as denser population areas leading to greater number of patients seeking care and relatively higher internet usage. As the physician got older (> 55 years old), they were also likely to get fewer reviews as compared to their younger counterparts [Figure 3]. This could be due to older physicians cutting back on practice and seeing fewer patients in addition to the younger physicians being more aware of the impact of PRWs and being more social media savvy.

Contrary to a study done by Baksh et al, wait time was close to significant in determining overall scores, while ease of making appointment did not correlate with overall score (9). Both physician bedside manner and physician/staff courtesy were more significant in correlating with the overall score of the physician. This not only suggests that physician bedside manner and courtesy have a greater impact on physician rating than wait time or ease of making the appointment, but also that customer service may play a greater role than the quality of physician care. Even if a patient waits longer than average but has a positive experience with the physician and staff, they are more likely to provide a positive rating. This further adds to the lack of consistency in criteria applied to rate physicians, thus bringing the validity into question.

Our study found that the majority of physician members of the ASES who responded to the survey held unfavorable views regarding the PRWs and the Affordable Care Act. Interestingly, even though we demonstrated that majority of the physicians are positively rated; they still do not regard PRWs highly. Physicians were concerned with “lack of validation”, “low sample size”, “lack of avenue to respond”, and “misrepresentation of their practice” reflecting what has been reported in the literature (10, 11). Furthermore, 74% of physicians reported to not regularly checking their rating profile, while 94% admitted to making no changes to their practice.

On the other hand, the minority that did benefit, made changes to improve their practice and believed it is an avenue to market their practice and get feedback from their patients. The overwhelming negative perception and abandonment of PRWs does not bode well for the future implication of PRWs on assessing quality based care. Furthermore, there is uncertainty whether more responses would change physician perception or the sample size that is sufficient and representative of their patient population. On the other hand, it could force physicians to pay attention to their profiles.

The validation process for physician rating websites remains controversial (11). A systematic review performed by Emmert et al reported PRWs have incomplete databases, low sample size, high risk of potential abuse due to anonymity, lack of validation, and inadequate evidence of actual patient outcomes (10). Most rating websites including Healthgrades.com and Vitals.com allow people to leave reviews anonymously, and though it serves to protect the identity of the person, it does bring uncertainty on whether the reviewer was even treated by that particular physician. An analysis of 33 PRWs, Lagu and colleagues found that less than 61 percent of these websites had a requirement for an email address prior to writing a review (1). Since the validity is questionable, the opportunity for potential fraud by disgruntled patients or competing practices is possible, while on the other hand, a practice may inflate their ratings by fabricating reviews. For example, a Minnesota neurologist sued one of his patients after the patient’s son posted a false review on a PRW. The neurologist then faced a hostile response with more than 30 fabricated negative reviews within two days. Finally, the Minnesota Supreme Court ruled that the freedom of speech right protects the patient in this scenario (12).

Most physicians and patients would probably agree that greater transparency in the publicly available information is needed. The Centers for Medicare and Medicaid Services (CMS) and American College of Surgeon (ACS) are implementing systems that would improve quality of physician rating (7). In the future, this system will guarantee compliance due to potential monetary penalties that are going to be imposed on physicians that do not participate. Furthermore, as measures such as quality reported by these websites are being considered in physician reimbursement, it is possible that these rating websites maybe influential in determining overall surgeon compensation (7).

There were several limitations to this study. We were not able to assess all ratings websites, though we did select the two most popular websites in Healthgrades.com and Vitals.com. In order to compare the rating scales, the 5-point scale of Healthgrades and 4-point scale of Vitals were converted into 10-point scales, which may have led to a slight discrepancy in the results. The study was only able to evaluate members of the American Shoulder and Elbow Society (ASES) and thus not reflective of the entire shoulder and elbow physician population. Furthermore, only 22% of the ASES membership responded to our survey. ASES membership entails a fairly extensive and rigorous membership process and thus might skew our data, There may be greater variability in online ratings and survey responses in the overall shoulder and elbow physician population. Although, there is no way to determine the average number of new patients seen by an orthopedic shoulder and elbow surgeon in a year, we did evaluate the average number of new patients seen per year at our home institution by a surgeon. The surgeons approximately see over a thousand patients a year. Therefore, the number of reviews merely reflects a 1-2% sample size of patients treated in a year by an ASES member. Furthermore, 58% of the reviews on Vitals.com had no comments, while only 10% were negative. This indicates that even fewer patients are providing context to support their rating.

As PRWs become increasingly popular amongst patients in this digital age, it is critical to understand that the scores are not reflective of a significant proportion of the physicians’ patient population. Patients’ positive reviews directly correlated with more with their “overall experience” in the physicians’ office. Surveyed physicians were seen to be unfavorable toward PRWs, the Affordable Care Act dealing with reimbursement, and the validity of their ratings. Physicians can use this study to determine what affects a patient’s experience and focus efforts on improving patients’ perception of quality, overall satisfaction, and overall care. Consumers may use this study to increase their awareness of the potential for significant sampling error inherent in PRWs when utilizing them to make decisions about their care.