What Patients Say About Their Doctors Online: A Qualitative Content Analysis

ABSTRACT

BACKGROUND

Doctor rating websites are a burgeoning trend, yet little is known about their content.

OBJECTIVE

To explore the content of Internet reviews about primary care physicians.

DESIGN

Qualitative content analysis of 712 online reviews from two rating websites. We purposively sampled reviews of 445 primary care doctors (internists and family practitioners) from four geographically dispersed U.S. urban locations. We report the major themes, and because this is a large sample, the frequencies of domains within our coding scheme.

RESULTS

Most reviews (63%) were positive, recommending the physician. We found a major distinction between global reviews, “Dr. B is a great doctor.” vs. specific descriptions which included interpersonal manner, “She always listens to what I have to say and answers all my questions.”; technical competence “No matter who she has recommended re: MD specialists, this MD has done everything right.”; and/or systems issues such as appointment and telephone access. Among specific reviews, interpersonal manner “Dr. A is so compassionate.” and technical competence “He is knowledgeable, will research your case before giving you advice.” comments tended to be more positive (69% and 80%, respectively), whereas systems-issues comments “Staff is so-so, less professional than should be…” were more mixed (60% positive, 40% negative).

CONCLUSIONS

The majority of Internet reviews of primary care physicians are positive in nature. Our findings reaffirm that the care encounter extends beyond the patient–physician dyad; staff, access, and convenience all affect patient’s reviews of physicians. In addition, negative interpersonal reviews underscore the importance of well-perceived bedside manner for a successful patient–physician interaction.

BACKGROUND

The Internet’s role in health care is growing rapidly. Increasingly, the Internet is used by patients to find medical facilities or doctors, to research specific medical conditions,^1–3 and to form support networks specific to health.^4,5 The Pew Internet & American Life Project 2008’s Tracking Survey found an increase in the number of Americans that look online for health information from 25% in 2000 to 61% in 2008.⁶

Among those who use the Internet for health care information, the majority (60%) access “user-generated” information, including reading others’ health experiences, and consulting rankings or reviews of health care facilities or health care providers.⁶ To date, Internet ratings of physicians have received much interest from media and doctors,^7–13 but little academic scrutiny. The one prior academic study of physician ratings, restricted to selected physicians in a small geographic area, revealed that Internet data on physicians is highly variable, with an absence of user-generated information on most.¹⁴

Because Internet reviews of physicians represent public perspectives from patients, they may provide valuable insights about patient perceptions of medical care. Therefore, we examined a wide range of publicly-available Internet-based reviews of primary care physicians, across different Internet sites and geographic areas, to better understand their content.

METHODS

Design

We conducted a qualitative content analysis of patient reviews about primary care physicians practicing in urban America. We chose urban areas because of the high density of doctors and the extensive availability of physician ratings from the Internet. We chose internal medicine and family medicine care because the majority of research on patient–physician interactions is focused on these primary care areas. We expect patient relationships with other primary care practitioners, such as obstetrician/gynecologists and pediatricians, to differ substantively. Similarly, relationships with subspecialty physicians would be likely to differ from those with primary care physicians.

Sampling

We employed a purposive sampling strategy in order to obtain a range of reviews of primary care physicians for adults, numerically balanced across four U.S. cities. We included reviews from Atlanta, Chicago, New York, and San Francisco as four cities from different geographical regions. We included internal medicine and family medicine physicians, excluding sub-specialists.

Data Sources

Our search strategy was meant to mimic two popular ways of searching for ratings using the Internet: 1) using a search engine and 2) using a well-known general ratings site. First, to mimic a patient’s approach, we utilized the popular Google search engine. When we entered the phrase “rate doctor” into Google.com, the first result was for the website ratemds.com. As its name suggests, ratemds.com exclusively rates physicians. Previous research has shown that there are at least 33 publicly available websites that allow patients to rate their doctors,¹⁴ and there may be more today. Second, because we surmised that patients might search for physician ratings on a website they use for other types of consumer ratings, we selected the website Yelp.com. We chose this general consumer and service rating site because of its ubiquity and large number of available ratings.¹⁵ Our aim was to have a balanced number of reviews from ratemds.com and Yelp.com.

Search Strategy

We obtained reviews from two websites: ratemds.com and Yelp.com. To obtain reviews from yelp.com, investigator (A.L.) entered “doctor” and the name of the city (e.g., “San Francisco”) into the search fields of the Yelp.com homepage. The website then generated a list of search results in the form of a list of doctors’ names (ranging between 200 and 690 doctors’ names, depending on location.) Each doctor had a varying number of reviews. Because we were only interested in general internists and family medicine physicians, we filtered our search to only show those doctors in the categories of “internal medicine” and “family medicine.” To search ratemds.com, investigator (A.L.) entered a zip code and then, using the website’s drop down menu, selected “internal medicine” and then “family medicine” (Fig. 1). We used ZIP codes for ratemds.com because at the time of the search, the website did not allow searches by city name. Importantly, all ZIP codes for a given city returned identical results.

Figure 1.

Method for selecting and including physician reviews.

Our sampling strategy had two distinct levels, because each physician could have multiple reviews. Each website first generates a list of physicians. Because the order in which doctors were listed on the website is non-random, we pre-specified our sampling of physicians as follows: We selected 30 reviews of doctors appearing at the beginning of the search results list, 40 reviews of doctors appearing in the middle of the search results list, and 30 reviews of doctors appearing at the end of the search results list. Next, we purposefully sampled the first three available reviews for each individual physician. We determined a priori that physicians with multiple ratings are likely to differ systematically from physicians with a single rating. Therefore, we included only up to three reviews for any single physician, to avoid bias introduced by selecting a large number of reviews from any particular physician. This allowed us to consider multiple reviews of the same physician independently of one another, with the assumption that each review represented a unique patient. No specific demographic information was available for reviewers on these websites.

Qualitative Analysis

We analyzed the data using content analysis.^16,17 Three investigators developed a coding template. In developing our coding template, we wanted to mimic the patients’ experience. Therefore, we conceptualized the medical appointment process into three sequential sections 1) prior to physician contact (calling to make the appointment, waiting for an appointment, interactions with staff, office environment, visit wait time) 2) face-to-face interaction with physician (patients’ perception of their doctor’s competence, communication skills, clinical skills) and 3) follow-up to encounter (physician’s follow up, referral, refills, patients’ overall satisfaction). Additionally, we searched the literature to include factors that have been associated with patient health outcomes (doctor’s friendliness, empathy) and added those to our list as well. Two investigators (A.L. and A.D.) added to and refined the coding template by independently coding 50 different reviews. When patients mentioned a factor outside of our original coding scheme, we added it to our coding scheme. For example, we did not think location of practice would become a theme in patient ratings, but we added it into our scheme because it emerged from the content analysis. Two investigators independently coded 328 (46%) reviews. We achieved high inter-rater reliability (kappa range 0.8-1.0 across codes); thus, the remaining reviews were coded by the lead investigator.¹⁸ We employed qualitative content analysis, utilizing both deductive and inductive reasoning.¹⁸ At the start of data collection we relied on concepts derived from our own medical experiences as well as patient satisfaction concepts found in the literature. As analysis of the data progressed, new concepts and variables of interest emerged from reviews. Our final coding scheme incorporated concepts and variables from all three areas. We knew saturation was reached when codes became redundant and no new codes were observed.¹⁹

The investigators reviewed and combined codes into larger themes by consensus. Thematic saturation was reached after approximately 100 reviews. All analyses were done using Atlas.ti software (Berlin, 1997). The investigators selected by consensus the most common themes for presentation in this paper.

RESULTS

Our search strategy resulted in 712 reviews (representing 445 doctors); 397 (56%) from ratemds.com and 315 (44%) from Yelp.com. By design, reviews were largely evenly distributed by city: Chicago, 28%; San Francisco, 27%; New York, 26%; and Atlanta, 19%. Upon use of the websites we found that most doctors appearing at the beginning of our search had a higher number of reviews and a higher overall rating score, while doctors appearing at the end of our search had fewer reviews and/or a lower overall rating score.

Reviews were first categorized as positive or negative. Next, in our analysis we made a clear distinction between global remarks and specific descriptions. Reviews categorized as global remarks consisted of general comments regarding the medical encounter or the doctor. In contrast, reviews that contained a more detailed account were categorized as specific descriptions and gave insight into the medical encounter or the doctor. Global statements were identified when a review lacked a specific description or action. For example a global review might be “My doctor is wonderful, kind, and caring.” Whereas a descriptive review might be “My doctor is wonderful, kind, and caring. He always listens to me and calls me to make sure I’m okay.” Or “My doctor is wonderful because I never have to wait long to see him and he never makes me feel rushed.” If investigators could answer the question “Why does the patient perceive this doctor positively or negatively?” then the review was specific. If however, the investigators could not identify why the patient perceived their doctor positively or negatively, then the review was global. Through sequential thematic analysis, three over-arching domains emerged within the reviews: interpersonal manner; technical competence; and system issues (Fig. 2). The interpersonal manner domain included items regarding the doctor’s characteristics, doctor’s attentiveness, and the doctor’s communication skills. The technical competence domain included issues relating to the doctor’s perceived aptitude, clinical skills, and follow-up or thoroughness. The system-issues domain included healthcare components outside to the patient–physician interaction including office staff, access to the doctor, and office environment.

Figure 2.

Predominant Factors in Themes Emerging from a Qualitative Content Analysis of Online Evaluations of Internal and Family Medicine Physicians for Four U.S. Cities in 2009.

The following tables provide examples of online reviewer comments by theme.

Global Remarks

Global reviews consisted of general comments regarding the medical encounter or the doctor (Table 1). They included comments about overall excellence, a recommendation of the doctor to others, expression of a negative sentiment, and few expressed intent not to return. Representative quotes include “My doctor is wonderful and the staff is nice,” and “He is wonderful.”

Table 1.

Global Remarks

Code	Representative Quotes	Count
Overall Excellence	Dr. B is a great doctor.	171
Recommendation	I love him and have recommended him to all my friends and family.	108
Negative Sentiment	I wouldn’t take my dog to him.	53
Intent not to return	I don’t have time to waste I will not go back.	27

Specific Descriptions

In contrast, reviews with specific descriptions were more detailed “She always listens to what I have to say and answers all my questions,” and “Dr. X listened to all my symptoms and eliminated all the unnecessary testing, I no longer feel like my entire life revolves around pain.” Three dominant themes emerged from specific reviews including interpersonal manner, technical competence, and system issues.

Interpersonal Manner

“Dr. X is friendly and caring.” Empathy was a central code mentioned under the interpersonal manner theme, both as a positive (69%) and negative (lacking in empathy, 31%). Negative reviews included, “Worst Dr. I’ve ever seen no concern for patients + arrogance off the charts…he really don’t give a !@#%.” Positive interpersonal reviews described doctors as empathetic, friendly, helpful, and trustworthy. Moreover, satisfaction with time spent with the doctor and feeling unhurried came up often in the interpersonal theme. Communication skills including listening and explaining also came up often in this theme with the majority of reviews expressing satisfaction. Reviewers also frequently mentioned how long they have been a patient of the doctor (Table 2).

Table 2.

Interpersonal Manner

Code	Representative Quotes (positive/negative)	Count	% Positive
Empathetic	Dr. A is so compassionate. / Not a people person.	258	69
Time spent with doctor during appointment(s).	Finally a Dr. who spends time with you and doesn’t watch the clock. /Towards the end of my physical I asked about a problem, and she said that would have to be discussed at another appointment.	161	75
Physician Listens	She listens to every issue I bring up. /This guy does not listen to what you have to say and hears what he wants to.	111	88
Longevity of relationship with physician	I have been with Dr. F for several years now.	106
Friendly	Dr. C is friendly.	102
Physician Explains	Takes the time to explain options for treatment./ Does not answer my questions	49	94
Nonjudgmental/Condescending	She makes embarrassing things not embarrassing./ Can have a snotty and demeaning attitude.	44	50
Helpful	Generally, helpful.	34	--
Trust worthy	I feel confident in his abilities and I have never seen a doctor who seemed so invested and devoted to his patients!	22	--
Puts patient at ease	She made me feel comfortable during the visit.	17	--
Competent	He seemed competent.	15	--

Technical Competence

Codes most frequently mentioned in the technical competence theme included a description of the doctor as knowledgeable, detailed and efficient. Examples include “Not only did he go the extra mile in time and stitches to ensure minimal scarring, he also put medical "glue" to further ensure a good closure,” and “She is very detail oriented and takes her time to write everything down.” Reviews also mentioned satisfaction with physicians’ follow up regarding diagnosis and test results as well as satisfaction with the physicians’ referrals to other providers. Negative technical reviews were more focused on perceived doctor error including perceived poor decision making. Representative quotes include “She mis-prescribed a medicine I have been taking for years. I have no intention of wasting my time with her again,” and “Didn’t take me seriously when I knew something was wrong with me. Turns out I had acute leukemia and she didn’t recognize my symptoms or refer me to a specialist.” (Table 3)

Table 3.

Technical Competence

Code	Representative Quotes (positive/negative)	Count	% Positive
Knowledgeable	He is always very knowledgeable.	114	--
Detailed	Very thorough.	94	--
Follows up	She was always on the ball with all her inquiries and test and test…and yes! She really does call with results!!!! / Never returns calls.	86	80
Clinical Skills	She asked me what I wanted to do what specialist I wanted to see as if I should know what to do with my health problems./ You can expect his course of treatment to be right on the dime.	60	52
Referrals	No matter who she recommends everything is right./ Sent me to a horrible GI.	55	78
Perceived Poor Decision Making	She mis-prescribed a medicine I’ve been taking for years.	40	--
Perceived Successfulness of Treatment	He listened, explained what I had and gave me a pill. Problem fixed./ He prescribed the wrong medication and permanently disfigured my (body part).	35	63
Efficient	Addressed my medical concerns in an efficient manner.	16	--
Complementary Alternative Medicine	He takes a holistic approach to health care and talks with me about alternative treatments.	14	--

System-Issues

The most common codes included staff, appointment access, appointment wait time, and practice characteristics including environment, health information technology, and location. For instance, “The office staff is a whole different can of worms. They are extremely hard to get a hold of in the first place (even during normal business hours) and seem like they are annoyed when you finally do get them on the phone.” Further, reviewers more commonly mentioned dissatisfaction with appointment wait time, “In addition, I usually spent between 30 and 60 minutes in his waiting room before I actually got to see him,” while they seemed more satisfied with their ability to make an appointment “I can’t say it’s easy to get in to see him all the time, but making an appointment is painless.” (Table 4)

Table 4.

System- Issues

Code	Representative Quotes (positive/negative)	Count	% Positive
Staff	The office staff remembers your name./Has a receptionist/nurse from hell.	175	60
Appointment wait time	His practice is busy, but I never have to wait for my appointment./I waited more than 45 minutes for each of the 2 appointments I had with her.	105	39
Appointment access	Appointments same day if warranted./ Forget seeing him when you are sick too, very inflexible that way. Will offer an appointment 2 weeks out.	69	57
Practice Office environment	Such a pleasant environment for my care./ I also find the clutter, dusty fake flowers, and especially her dog (!!!) to be inappropriate for a doctor’s office.	50	56
Method of MD selection	My mom recommended her.	45	--
Cost of Care	This doctor took blood test that came to over 3000 dollars in cost.	35	--
Negative view of health care	I hate doctors.	21	--
Practice Health Information Technology	He came in with his apple book and took notes while I talked.	20	--
Practice Location	His practice is just down the block./I drive an hour to see her, but it is worth it.	14	--

Discussion

Patient review sites provide anyone the opportunity to review a doctor in an anonymous, self-driven, and unstructured format. The sites are free of cost, and provide site users an opportunity to post their unrestricted comments. Thus, they differ significantly from the traditional patient satisfaction surveys with which health systems and researchers typically assess patient satisfaction. First, the anonymity of Internet reviews confers both drawbacks and benefits. Patients are able to evaluate their doctors without worrying that criticism might change the quality of care they receive; conversely, anonymity leads to a lack of accountability towards physicians by their reviewers. Second, traditional satisfaction assessments are distributed to patients by health systems or researchers and assessments are limited to a single encounter or time frame. In contrast, online reviews allow patients to take the initiative when providing feedback, rather than being asked to complete a patient satisfaction form at a particular time. Internet review sites are independent of health systems and patients may perceive them differently than a patient satisfaction questionnaire coming from a physician’s office. Third, results from traditional patient satisfaction questionnaires are pooled, while a single individual Internet review can have more influence than an individual patient’s survey responses. Fourth, online reviews are unstructured, whereas traditional patient satisfaction surveys are often close-ended questions. Thus, issues not addressed in closed-ended patient satisfaction instruments may come to light in Internet reviews. Although Internet ratings of physicians remain incomplete, with a recent study finding that the majority of local physicians did not have reviews online,²⁰ they are rapidly expanding, and becoming increasingly important. Because Internet ratings are unfiltered, public, and consist of individual rather than aggregated ratings, many physicians perceive a danger to their professional reputations.^8,9,11 Our data suggest that most, if not many, Internet reviews espouse positive sentiments about primary care physicians.

Although global reviews did not provide information on the drivers of satisfaction and dissatisfaction, specific reviews could provide fresh insights. From these, we note similar patterns to prior patient satisfaction research,²¹ as follows. We found that trust and confidence in doctor,²² ease of appointment making²³ and a good interpersonal relationship²⁴ lead to patient satisfaction among Internet reviewers as well as in other settings.

We know that currently used patient satisfaction measures do not capture all patient perceptions; some patients are much more likely to complete a satisfaction survey- or complain- than others.²⁵ Further, measuring patient satisfaction is multidimensional and therefore results must be interpreted with consideration²⁶. A review of 195 studies using patient satisfaction instruments found little evidence of reliability or validity.²⁷ Although the evidence is mixed, previous studies of traditional satisfaction assessments reveal respondents to be older and less educated than non-respondents.^25,28 Because patients completing Internet reviews are younger and more affluent, we suspect that Internet reviews may better capture the subset of patients less likely to respond to traditional patient satisfaction assessments, and thus may inform our understanding of the drivers of patient satisfaction for younger and educated patients. It is also possible that widespread public availability of patient opinions of physicians may affect the way patients and physicians interact. Moreover, Internet reviews do reflect certain aspects of the patient–physician relationship that are known to relate to health behaviors. For example, perception of the doctor as empathetic and friendly, which was a common theme in Internet reviews, has been associated with adherence.²⁹ In addition, satisfaction with the doctor’s communication skills, also found in these reviews, has been shown to improve patient satisfaction³⁰ and adherence.³¹ Interestingly, patients did not tend to list specific qualifications, such as the prestige of their medical school or post-graduate training, in recommending their physicians.

Patient dissatisfaction appears from prior work to be more than the absence of satisfaction and has been associated with factors including nonsocial doctor²¹ as well as difficulty making appointments²¹ and longer wait time.³² Similarly, we found that the themes of antagonistic physician, dissatisfaction with wait time, and difficulty making appointment makings were frequent in negative Internet reviews.

The subset of specific, negative reviews may be particularly useful for providers and health systems. First, negative interpersonal reviews underscore the importance of well-perceived bedside manner for a successful patient–physician interaction. In addition, our findings reaffirm that the care encounter extends beyond the patient–physician dyad; staff, access, and convenience all affect patient’s reviews of physicians. Such reviews could prompt efforts to make the office environment more patient-centered.

Nevertheless, our study has several limitations. First, demographic information is very limited. Based on the information available, it appears that persons who write Internet reviews of physicians are likely to differ from health care consumers in general. In particular, they are likely to be younger, healthier, and more affluent than health care users overall. Yelp.com maintains some demographic information on those that visit and subscribe to their site. The highest proportion of online reviewers are young adults between the ages of 18-34 (46%) college graduates (47%); 35% make more than $100, 000 a year.³³ Comparable information was not available from RateMDs.com, because they do not collect demographic information about their website’s online reviewers. This complements the oft-cited response bias of traditional patient satisfaction surveys, in which more frequent health care users are more likely to respond.^34,35 Second, we could not determine the exact algorithm for the order in which search results appear. We attempted to account for this potential bias by pre-specifying a purposive sampling strategy across the range of search results (to include physicians appearing at the top, middle, and bottom of search results). We also included only three reviews per physician. Including three reviews per physician limits our ability to draw conclusions for any individual physician, because we may not accurately capture the breadth of opinions about that individual. However, we judged it more important to avoid over-weighting our results towards characteristics of frequently-reviewed physicians because we expect these to differ substantively from seldom-rated physicians. Our finding that positive results were more likely to appear at the beginning of the search supports this sampling approach. Third, we restricted our results to urban locations in order to have sufficient density of physician reviews; rural areas are likely to differ systematically. Fourth, we limited our study to internal medicine and family medicine. Patient relationships with obstetrician/gynecologists may be confined to a short duration (such as pregnancy) and may not be a primary care relationship.^36,37 For pediatricians, the relationship is between the parent(s), patient, and provider³⁸ and this differs substantively from the one-on-one relationship between adults and their internal or family medicine providers. Thus we are hesitant to generalize our results to other primary care disciplines or to medical specialties.

Internet reviews of primary care physicians provide unfiltered patient perspectives on health care; as such, they provide useful insights into improving the patient-doctor relationship. It is not clear how medical practice will change with the advent of widespread, publicly-available physician reviews. The possibility of receiving a public review may alter communication between doctors and patients, and thus change the doctor-patient relationship. Online reviews of doctors could change patient–physician communication, because online reviews shift the balance of authority in the relationship. Patients are now able to evaluate their doctor in a public forum with complete anonymity. Often, physicians cannot respond to concerns raised in Internet reviews without breaching patient confidentiality standards. These public, unedited, potentially inaccurate reviews can later be accessed by potential new patients and future employers, impacting a physician’s career and reputation. Additionally, in an effort to protect themselves, doctors may have patients sign mutual privacy agreements or other similar agreements in an effort to mitigate potential criticism^39–41. Although, we cannot predict the exact effect, we do expect these reviews to influence trust and communication in the patient–physician relationship. Future studies should explore physician reactions to Internet reviews and observe their effect on medical practice and communication.