Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Nov 1.
Published in final edited form as: BMJ Qual Saf. 2015 Dec 15;25(11):889–897. doi: 10.1136/bmjqs-2015-004515

From the closest observers of patient care: A thematic analysis of online narrative reviews of hospitals

Naomi S Bardach 1,2,5, Audrey Lyndon 3, Renee Asteria-Penaloza 2, L Elizabeth Goldman 4, Grace A Lin 2,4,5, R Adams Dudley 2,4,5
PMCID: PMC5360532  NIHMSID: NIHMS844152  PMID: 26677215

Abstract

Objective

Patient-centred care has become a priority in many countries. It is unknown whether current tools capture aspects of care patients and their surrogates consider important. We investigated whether online narrative reviews from patients and surrogates reflect domains in the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) and we described additional potential domains.

Design

We used thematic analysis to assess online narrative reviews for reference to HCAHPS domains and salient non-HCAHPS domains and compared results by reviewer type (patient vs. surrogate).

Setting

We identified hospitals for review from the American Hospital Association database using a stratified random sampling approach. This approach ensured inclusion of reviews of a diverse set of hospitals. We searched online in February 2013 for narrative reviews from any source for each hospital.

Participants

We included up to two narrative reviews for each hospital. Exclusions: Outpatient or emergency department reviews, reviews from self-identified hospital employees, or reviews of <10 words.

Results

50.0% (n=122) of reviews (N=244) were from patients and 38.1% (n=93) from friends or family members. Only 57.0% (n=139) of reviews mentioned any HCAHPS domain. Additional salient domains were: Financing, including unexpected out of pocket costs and difficult interactions with billing departments; system-centred care; and perceptions of safety. These domains were mentioned in 51.2% (n=125) of reviews. Friends and family members commented on perceptions of safety more frequently than patients.

Conclusions

A substantial proportion of consumer reviews do not mention HCAHPS domains. Surrogates appear to observe care differently than patients, particularly around safety.

Keywords: Hospitals, quality indicators, health care, patient participation, patient satisfaction, qualitative research

BACKGROUND

Countries increasingly are including patient experience measures in national public reporting or payment programs.1–6 In the US, the Center for Medicaid and Medicare Services (CMS) uses the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) patient experience survey in payment calculations.7,8 Hence, health system leaders are increasingly asking clinicians to review and respond to these data.7,9 While HCAHPS covers 8 domains (e.g., nurse and doctor communication, pain control, and discharge information), the measure was developed in 2003–2004 and it is unknown whether those domains reflect the aspects of care that are most important to contemporary patients. Since performance measurement tied to public reporting or incentives may direct attention away from other aspects of care,10–12 it is critical that measures of patient experience provide a comprehensive assessment of the aspects of care most salient to patients and their surrogates.

Narrative online hospital reviews provide a relatively popular and novel source of quality information from and for patients,13–18 and represent what patients value and observe about care when not offered structured prompts. Furthermore, the perspectives of friends and family members are increasingly being recognized as important to patient outcomes,19–21 but are not captured in patient-only surveys such as HCAHPS. In addition, prior research suggests that online numerical ratings accompanying narrative reviews are associated with patient outcomes such as mortality, readmissions, and infection rates.1,22,23 While further investigation is warranted, this suggests that the patient and friend and family member reviewers may observe care that is related to these outcomes. Lastly, these online data provide access to assessments of a range of geographically and organizationally diverse hospital types.

Prior analyses have described themes in online reviews of urban outpatient providers,24,25 emergency department visits,26 in hospital-related microblogs on Twitter,27 and in pre-prompted narrative reviews of NHS hospitals in England.28 There are no published data assessing whether unprompted consumer criteria for hospital quality are similar to domains in structured surveys; nor data assessing narratives from friends and families, separate from patients. Lastly, the original HCAHPS developers chose survey domains based on family and patient focus groups over a decade ago29,30 and new domains of importance to patients, friends, and families may have emerged with subsequent healthcare system changes. We performed this study to meet the following research objectives: to assess whether the HCAHPS domains are salient to contemporary consumers, whether there are additional domains that might be emerging in a changing healthcare environment, and whether surrogate perspectives may differ from patient perspectives.

METHODS

Overview

To achieve the goals of the study we qualitatively analysed a group of online narrative reviews from a diverse sample of US hospitals. Online reviewers numerically score hospitals and provide accompanying narratives. Our underlying premise is that the narrative stories are told for a reason and provide context and justification for their accompanying quantitative score. These narratives therefore provide a datasource to understand which domains online reviewers use when judging hospital quality.

In order to analyse the narratives for already-defined HCAHPS domains as well as domains that were not included in HCAHPS but which may be important to consumers, we chose the qualitative method of thematic analysis. This is a flexible approach described as: “searching across a data set – be that a number of interviews or focus groups, or a range of texts – to find repeated patterns of meaning.”31 Key themes are identified both through an assessment of prevalence of the themes within the text as well as salience to the research question at hand.31 In addition to our qualitative analysis, we include a quantitative approach to check our assessment of theme prevalence as well as to check our impression that themes differed by reviewer type.

Sample

To create our sample of narrative reviews, we defined a national group of hospitals from the American Hospital Association database, which includes hospital characteristics for all acute care hospitals in the United States (n=6335). We chose to limit our sample to the US given the heterogeneity of healthcare environments worldwide and because HCAHPS was developed originally in the US context. We stratified the group using the variables the Agency for Healthcare Research and Quality (AHRQ) HealthCare Utilization Project uses to create the Nationwide Inpatient Sample, a nationally representative administrative dataset: region of the country, urban and rural location, teaching status, and size.32 We randomly sampled hospitals within these strata.

We searched for reviews on the sampled hospitals. In prior work, we found that certain hospital characteristics (high bed number, Western region, and private not-for-profit status) were associated with having more online narrative reviews, with some hospitals with >100 reviews.22 In order to avoid oversampling reviews from hospitals with these characteristics we took 2 reviews from each hospital we sampled, which we found in prior work to be the median number of reviews per hospital.22 Based on a pilot analysis of 50 reviews33 we anticipated that approximately 250 inpatient reviews would result in thematic saturation.

In order to define our search strategy for online reviews we tested search terms and search engines to optimize our results. For each new hospital search we cleared our prior search history, to prevent this from affecting results for subsequent searches. We used an iterative approach to sample hospital reviews. From among the reviews found within the first five pages of search results, we used the two most recent reviews because people have changed their willingness to review healthcare providers within the last few years.34,35 We excluded: emergency department (ED) reviews, because the ED and inpatient setting differ substantially; review sites with both structured items and unstructured narrative reviews, since structured items suggest domains to reviewers; and reviews that were less than 10 words long. See Appendix for additional details on the above processes.

Analysis

As noted above, we used thematic analysis31 and assumed that reviewers wrote with the intention to convey their experience.36 For instance, a review describing a medical error indicates that the reviewer uses safety as a quality criterion, regardless of whether the event was an error according to accepted medical standards. In order to assess for delineated HCAHPS domains as well as additional themes, we read iteratively and openly and created a shared codebook for domains relevant to the research question.31 The thematic analysis approach allows for both the approaches we use here: reading and coding data based on existing literature; as well as reading and coding for themes that emerge from the text that are salient to the research question. This flexible analytic approach focuses on key themes relevant to the research question while still maintaining an openness to new concepts in the data.31

The research team met to discuss the HCAHPS domains, reviewing the survey items that comprise each domain. Two coders (XXX and XXX) created a set of codes and memos representing the HCAHPS domains. We created an additional code “General communication” for global or not person-specific comments about communication (e.g., “the staff was very comforting in their words”), since these could not be coded as HCAHPS nurse- or doctor-specific communication domains. Comments about communication with specific non-clinicians (e.g., security guards) were not included in this code. We also developed codes describing reviewer types, diagnostic category, whether paediatric or non-paediatric, and positive or negative valence. See Table 1 for coded categories.

Table 1.

Review Characteristics (n=244)

Reviews, n (%) Average star rating given by reviews in that category (sd)
Reviewer Type
 Patient 123 (50.0) 3.1 (1.8)
 Friend/Family Member 93 (38.1) 3.2 (1.8)
 Unable to determine 28 (11.9) 2.6 (1.7)
Service*
 Medicine 47 (19.3) 3.1 (1.8)
 OB/Gyn 40 (16.4) 3.7 (1.7)
 Surgery 27 (11.1) 2.5 (1.7)
 Paediatrics 23 (9.4) 3.1 (1.9)
 Mental health 5 (2.1) 2.6 (2.2)
 Not Specified 112 (45.9) 2.8 (1.8)
Valence
 Positive 117 (48.0) 4.7 (0.6)
 Mixed 15 (6.2) 3.5 (0.9)
 Negative 112 (45.9) 1.3 (0.8)
Open in a new tab
*

Some reviews were for more than one service line, so total is greater than 100%.

Before analysing study data, the primary coders (XXX and XXX) coded several sets of older reviews from an online review site. The coders then met to discuss areas of concurrence and discrepancy to develop a consensus approach before additional analysis. The coders then analysed the primary data set separately by using the HCAHPS domains codes and developing new codes for additional domains. The coders met regularly to review coding and reconcile differences.

Others experienced in qualitative analysis (XXX, XXX, XXX, XXX) read all reviews and refined the codebook in an iterative fashion. We maintained methodological rigor using this triangulation, with multiple analysts and an interdisciplinary team.37 Clinical investigators’ backgrounds were obstetric nursing (XXX), pulmonary critical care (XXX), internal medicine (XXX and XXX), and paediatrics (XXX). XXX provided a non-clinician, consumer interpretation. Each clinical researcher had been hospitalized personally and/or had at least one close family member hospitalized, for obstetric, medical, or surgical admissions and intensive care unit admissions.

We generated descriptive statistics of the consumer reviewers’ characteristics and the proportions of reviews mentioning each HCAHPS domain. We used Chi-squared analysis, or Fisher’s exact test for cell sizes <10, to assess relationships between specific domains and reviewer type.

No identifiable patients were included. The study was considered exempt by the institutional review board of the University of California San Francisco. We report our findings in accordance with the COREQ qualitative research reporting guidelines (see Appendix for COREQ checklist).38,39

RESULTS

Review Characteristics

Our sampling strategy resulted in a total of 244 reviews within our sampling strata across 193 hospitals (3% of the hospital sampling universe, with balanced groups across strata). In the iterative sampling process’ 38 hospitals had no reviews and were replaced with another hospital in the same stratum. Investigators agreed that we reached thematic saturation after reviewing the corpus of reviews. The websites that were the most common sources of reviews were Yelp.com (32.0%, n=78), plus.Google.com (18.9%, n=46), and local.Yahoo.com (16.4%, n=40).

Half of reviews were from patients and 38.1% from friends or family members; in the remainder, we could not determine whether the reviewer was a patient or friend or family member (Table 1). The most commonly described diagnoses were in medicine and obstetrics; however, almost half of reviews did not specify the reason for admission. Similar proportions of reviews were negative (48.0%) and positive (45.9%), with few mixed reviews (6.2%). The average star ratings (which were all out of 5 stars) were moderately positive for patients (3.1, standard deviation (SD): 1.8) and friends or family members (3.2, SD: 1.8). Average star ratings differed by service (Table 1), as has been noted in prior work on patient experience ratings.40

HCAHPS Domains

Fifty-seven percent of reviews mentioned an HCAHPS domain (Table 2). The most frequently mentioned HCAHPS domains were communication with nurses (23.8%) and with doctors (26.6%). Few reviews mentioned pain control (7.4%), discharge information given (7.4%), or communication about medicines before administration (4.9%). Including “General communication” increased the percentage of reviews mentioning any HCAHPS domains by 6.5%, with the remaining 36.5% not mentioning any.

Table 2.

Domains mentioned by reviewer type

All reviews (n=244), % Reviewer type
Friend or family member (n=93), % Patient (n=123), %
HCAHPS domains
 Communication with nurses 58 (23.8) 25 (26.9) 32 (26.2)
 Communication with doctors 65 (26.6) 25 (26.9) 35 (28.7)
 Responsiveness 40 (16.4) 20 (21.5)* 20 (16.4)*
 Environment** 34 (13.9) 15 (16.3) 19 (15.6)
 Pain control 18 (7.4) 6 (6.5) 11 (9.0)
 Medicine communication 12 (4.9) 6 (6.5) 6 (4.9)
 Discharge information 18 (7.4) 10 (10.8) 8 (6.6)
No HCAHPS domain mentioned+ 105 (43.0) 32 (34.4) 49 (40.2)
Non-HCAHPS domains
 Patient safety 101 (41.4) 49 (52.7)++ 43 (35.0)++
 Financing 28 (11.5) 6 (6.5) 15 (12.2)
 System-centred care 19 (7.8) 4 (4.3) 12 (9.8)
Open in a new tab
*

p<0.05, using Fisher’s exact test for cell sizes smaller than 10

**

This included items regarding room and bathroom cleanliness and room noise level at night

+

If including “General Communication”, 36.5% of all reviews have no HCAHPS domains coded.

++

p<0.01, using Fisher’s exact test for cell sizes smaller than 10

Non-HCAHPS Domains of Patient Experience

In contrast, 51.2% of reviews contained themes that are not included in HCAHPS. We report key themes that were salient to the research goal–to contribute to the literature on understanding patient-centred care in order to improve care systems. We defined key themes as those that at least 5% of reviews mentioned (for a minimum definition of prevalence)31 and that were potentially actionable (for additional themes see Appendix). These were: Financing, including unexpected costs and difficult interactions with billing departments; System-centred care; and Perceptions of safety. We describe the themes below, and provide illustrative quotations. Friends and family members were more likely to comment on perceptions of safety (n=49, 52.7%) than were patients (n=43, 35.0%), without differences in other themes by reviewer type (Financing p=0.17, System-centred care p=0.19).

Financing: unexpected costs and difficult interactions with billing departments

This theme emerged in 11.5% (n=28) of the reviews. Reviewers commented on the variation in pricing for similar services, confusion about billing practices, and the stress of dealing with billing departments and unexpected out-of-pocket costs. A call for greater price transparency stemmed from a desire to avoid unplanned costs.

“[…] I had a CT Scan and it was $5300. I had one done 6 months later at a hospital in St. Louis and it was $2100…same test, read by the same type of doctor. ?!?! […]”

“Good service, big bill on specialist. Your insurance may not cover most of it. […] Call the hospital and ask for the cost first before you commit into any specialty sessions […] ”

“I tried to pay on my bill before my insurance made payments and they would not accept them and after insurance paid they were rude and demanding. According to them $50 a month was not enough and they wanted almost $400 a month to keep from turning me into collections […]”

System-centred care

Some reviewers experienced the hospital system as disempowering, delivering system-centred, not patient-centred, care. In these comments (7.8%, n=19), reviewers describe experiences resulting from a focus on enforcing system policies, rather than a patient-centred approach.

“Just recently my son had a seizure. They scheduled us an EEG […] the procedure required my son to be sleep deprived so he only got 3 hours of sleep. I had never been to that hospital and […] check in was at 900. I got there at 915 […] 5 mins later she calls me up to the desk to tell me the tech performing the procedure has left to do his other appointments. I became very upset my son is only 12 and is at this point exhausted. Then the tech calls there to talk to me and tells me I was supposed to be there at 830 […] the procedure was not even until 930 and he left 5 min before that”

The quote above illustrates how poor communication and inflexibility in system policies (which expect patients to arrive early for appointments, but give mixed messages as to when, and in which arriving 15 minutes after check-in evokes a punitive experience—the technician physically leaves the premises, making the situation nearly irreversible) can lead to frustration for patients and families. This quote also illustrates the confusion some patients experience around expected arrival time compared to appointment time, which could convey a message to patients that their time is less important than the overall functioning of the system.

Conversely, patients who experienced the system as patient-centred, rather than system-centred, discussed how all aspects of the system (doctors, nurses, nutrition services, etc.) supported the patient’s needs, rather than the patient feeling that his or her needs are secondary to the system needs:

“[…] From the moment I stepped in the door, to the wheelchair ride at the end of the visit, I felt like the most important person there. No waiting for doctors, nurses, meds, meals periodic checkups […]”

Perceptions of safety

This was a theme in 41.4% of the reviews, Subthemes were: trust in specific providers, unsafe practices or outcomes, and observations of missed or incorrect care. In a sensitivity analysis, each of these subthemes was commented upon more frequently by friends and family members than by patients (data not shown).

Trust in specific providers: “At one point, our family member had inappropriate bleeding from several sites on the body which [a doctor] dismissed as unimportant. Eventually we were able to convince other people in the ICU to take care of the situation.”

Unsafe practices or outcomes: “My other concern has to do with […] following standard sanitary procedures for infections, masks, gloves, throwing stuff away that is needed and then fetching it out of the trash, not having dressing kits on hand.”

“The only thing that worries me about this place is that alot of people seem to come back with c-diff and or infections”

Observations of missed or incorrect care: “My family member was mostly ignored, […] They brought [her] a tray of food 6 times, despite being told (by us!) that she was unable to eat because she was about to have testing done. […]”

While additional reviews in our sample expressed a general distrust of the system, the reviews we included in this category were anchored in specific observations of care around safety or trust. This theme was mentioned more often (52.7%, n=49) in friend and family reviews compared to patient reviews (35.0%, n=43; p<0.01 for comparison).

DISCUSSION

In this study of online narrative reviews from a national sample of US hospitals, we provide new insights regarding consumers’ unprompted perspectives on aspects of hospital quality compared with structured quality assessments. We found that while communication was mentioned often, there were some HCAHPS domains frequently missing in reviews, and that there were additional non-HCAHPS domains raised by patients, friends, and family members. We found that family members and friends represented a large proportion of online reviewers. Our findings suggest that these surrogates may differ from patients in which aspects of care they use to define quality.

Strengths of this study include the sampling approach, which used reviews from all online sources, not restricting our sample to one website, and ensured a diverse sample of over 100 hospitals. Prior studies of online reviews have used geographically limited samples of hospitals and have been restricted to only one or two sources of reviews (e.g., Yelp.com, rateMDs, jameda).3,24,25,28 Prior studies have also not analysed themes by reviewer type (patients vs. friend or family member). Lastly, most prior studies of online narrative reviews analysed reviews for physicians, not hospitals. Though Lagu et al. studied online hospital narrative reviews, consumers were prompted first to answer structured domain items and the sample was NHS hospitals in England.28 Ours is the first study analysing unprompted narrative inpatient reviews in a multi-payer environment, in which reviewers may be at-risk of financial burden from excess medical costs.

Our findings are important for clinicians and hospital leaders for two reasons. First, a focus on HCAHPS scores, or other patient experience surveys with similar domains, may direct attention to issues that are of lesser importance to consumers, at least as judged by the rarity with which consumers mention them in reviews. Second, since there appear to be topics that are important to consumers but about which data is not collected, clinicians and hospitals risk achieving low patient experience scores without knowing that performance in un-queried domains may need to be improved. In particular, the theme “Perceptions of Safety” indicates that patients and family members are a potential untapped source of information on patient safety.41 Since explanatory, detailed narratives from patients may be more actionable than quantified measures, clinicians and hospital administrators may be interested in using their content to guide quality improvement.42 Because reviews accumulate with time, providers and policy-makers may find it worthwhile to monitor narrative reviews for emerging issues, relevant to individual hospitals or to the broader healthcare system.

While HCAHPS is a well-validated measure of patient experience,29,40,43–46 three of its eight domains – pain control, communication about medicines, and discharge instruction – appeared in less than 10% of reviews. This echoes findings from the formative HCAHPS focus groups, in which participants, when pressed, preferred information on the other domains compared to pain management, avoiding medication problems, and avoiding problems after discharge.44 However, these domains were still considered important to most focus group participants, and so were included in the HCAHPS survey.44 None of the key domains we describe in this study arose in the original focus groups except hospital-acquired infections, a sub-theme within safety. Potential reasons for different themes in the two studies include different ways of eliciting responses (focus group vs. online reviews), changes in the US healthcare environment, or different populations. It is possible that the broader safety theme may have been an under-recognized theme in the developmental work, at a time when expectations were lower that patients and surrogates could observe aspects of safety. Additional work remains to assess whether the other domains we found, if measured, could better describe patients’ perceptions of their experience, and could be a better gauge of whether a hospital is providing patient-centred care. Including additional domains would increase cognitive burden for respondents and data collection expenses, and so new HCAHPS items would need to be carefully evaluated to assess validity prior to inclusion.

Potential areas of focus for those hoping to make hospitals more patient-centred are suggested in the non-HCAHPS domains we describe:

Financing

There is increasing awareness of differences in prices across hospitals and emergency departments,47 and a desire for price transparency, as reflected in the reviews. This topic has become more salient in the US now than in 2003–2004 when the HCAHPS focus groups convened,48 with more US insurance plans incurring high out-of-pocket consumer costs,48,49 but is relevant to patients with private insurance worldwide.50 This study provides data documenting that frustrations with price variation and opaque billing processes frequently lead to negative experiences and negative public postings. Clinicians and hospital administrators could consider monitoring and improving this experience for patients. Further research also could assess the relationship between health outcomes and the financial sequelae from the index hospitalization, given known associations between financial well-being and better health.51

System-centred care

Consumers can encounter administrative barriers to care and experience bureaucratic disempowerment in interactions with the hospital system. This domain is conceptually similar to a set of items Rothman et al. added to HCAHPS, which they found were more predictive of patient willingness to recommend a hospital than many of the original HCAHPS domains.52 It is possible that the sense of disempowerment may also contribute to decreased patient activation,53 which has been associated with worse outcomes, including increased readmissions, and higher costs.54–58 Gathering data on patient experiences in navigating the hospital systems of care may help hospitals improve their overall patient-centeredness, and the resulting impact on patient activation and patient outcomes should be evaluated.

Perceptions of safety

Safety or loss of trust arose as an important theme in more than 40% of consumers’ narrative reviews. Evidence that patient experience ratings are associated with hospital-level rates of nosocomial infections and in-hospital mortality,1,22 taken together with our findings, suggests that patients or caregivers may be a useful source of information about patient safety. Their observations may be particularly important given stubbornly high rates of failures in patient safety.41,59–62 Gathering patient reports on safety has been suggested in the past,41 but has focused on gathering specific incident reports when they happen, rather than a global, standardized, and regular assessment from all patients. While discrete process and outcomes measures on patient safety exist, a global assessment of perceptions of safety would overcome one limitation of disease-specific measures, which by design are applicable to only a small percentage of hospitalized patients.

Finally, it is not currently standard to survey family and friends regarding the experience of a patient’s hospitalization. The finding that family and friends’ reviews included safety and trust themes more frequently than patients’ implies that this group could provide useful data in these areas. Next steps include developing methods for gathering data from patients, friends and family members on perceptions of safety, and assessing whether acting on these data can lead to improved health.

Limitations include a potential response bias, if perspectives of online reviewers differ from those of the general population. Internet users have been noted to be younger and less sick compared to others.34,63 It is possible that this response bias affects the themes, as it has been shown that quality topics of interest differ by age in a group of online visitors to public reporting websites.64 However, our sampling is more geographically diverse than prior studies, including the formative focus groups in the HCAHPS development;65 and the new topics we found, even if they are more salient to one subpopulation of consumers than another, are nonetheless important on face value and for the groups that respondents do represent. Another potential limitation is that people with extreme feelings may be more likely to write unsolicited comments, resulting in a portrayal of quality themes that may differ compared to the general public. However, this is also a limitation of in-person focus group data, which is the other potential source of quality domains in hospital care.65,66 In addition, most of our research team has clinical training (MD or RN) and personal or family experiences in the hospital, with one non clinician coder. This represents a strength in recognizing themes that may be actionable for creating a more patient-centred system. However, clinical training may also influence our analysis toward clinical rather than non-clinical domains. In part to address this limitation, we provide the full list of domains from the analysis in the Appendix. Lastly, we sampled reviews from US hospitals, and so findings may not be generalizable to other countries. For countries with public or mostly public insurance models, the findings on financing, though less applicable, may be useful for policy discussions regarding potentially increased privatisation.

CONCLUSION

In conclusion, this analysis of a national group of online hospital reviews suggests that a widely used tool for measuring patient experience may not adequately capture some potentially important domains, including perceptions of safety and experiences of difficulties with billing and unexpected out of pocket costs. Our findings also suggest that friends and family members’ perspectives on patient safety differ from patient perceptions and may provide additional important information.

Supplementary Material

Appendix

Acknowledgments

Study sponsors were not involved in design, data collection or analysis, writing of the report, nor decision to submit the article for publication. All authors had full access to the data and take responsibility for data integrity and accuracy of the analysis.

Footnotes

Researchers have no conflicts of interest to disclose and were independent from the funders.

References

  • 1.Greaves F, Pape UJ, King D, et al. Associations Between Web-Based Patient Ratings and Objective Measures of Hospital Quality. Arch Intern Med. 2012 Mar 12; doi: 10.1001/archinternmed.2011.1675. [DOI] [PubMed] [Google Scholar]
  • 2.Emmert M, Meier F, Heider AK, Durr C, Sander U. What do patients say about their physicians? an analysis of 3000 narrative comments posted on a German physician rating website. Health Policy. 2014 Oct;118(1):66–73. doi: 10.1016/j.healthpol.2014.04.015. [DOI] [PubMed] [Google Scholar]
  • 3.Emmert M, Meier F. An analysis of online evaluations on a physician rating website: evidence from a German public reporting instrument. J Med Internet Res. 2013;15(8):e157. doi: 10.2196/jmir.2655. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.The Patient Experience of Care Domain in Hospital VBP. 2015 http://www.medicare.gov/HospitalCompare/Data/Patient-Experience-Domain.html. Accessed May 7, 2015.
  • 5.Black N, Jenkinson C. Measuring patients’ experiences and outcomes. BMJ. 2009:339. doi: 10.1136/bmj.b2495. 2009-07-02 11:07:16. [DOI] [PubMed] [Google Scholar]
  • 6.Coulter A, Locock L, Ziebland S, Calabrese J. Collecting data on patient experience is not enough: they must be used to improve care. BMJ. 2014:348. doi: 10.1136/bmj.g2225. 2014-03-27 00:06:26. [DOI] [PubMed] [Google Scholar]
  • 7.HCAHPS: Patient’s Perspectives of Care Survey. 2013 http://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/HospitalHCAHPS.html. Accessed September 5, 2014.
  • 8.HCAHPS Survey Instruments. 2014 Sep 9; 2014; http://www.hcahpsonline.org/surveyinstrument.aspx.
  • 9.Fiscal Year 2009 Measures for Inpatient Quality Reporting Program 2010 Payment Update. 2013 https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/HospitalQualityInits/Downloads/HospitalRHQDAPU200808.pdf. Accessed September 9, 2014.
  • 10.Bardach NS, Cabana MD. The unintended consequences of quality improvement. Curr Opin Pediatr. 2009 Sep 21; doi: 10.1097/MOP.0b013e3283329937. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Omoigui NA, Miller DP, Brown KJ, et al. Outmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes. Circulation. 1996 Jan 1;93(1):27–33. doi: 10.1161/01.cir.93.1.27. [DOI] [PubMed] [Google Scholar]
  • 12.McDonald R, Roland M. Pay for performance in primary care in England and California: comparison of unintended consequences. Ann Fam Med. 2009 Mar-Apr;7(2):121–127. doi: 10.1370/afm.946. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Huppertz JW, Carlson JP. Consumers’ use of HCAHPS ratings and word-of-mouth in hospital choice. Health Serv Res. 2010 Dec;45(6 Pt 1):1602–1613. doi: 10.1111/j.1475-6773.2010.01153.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014 Feb 19;311(7):734–735. doi: 10.1001/jama.2013.283194. [DOI] [PubMed] [Google Scholar]
  • 15.Social media “likes” healthcare: From marketing to social business. 2012 http://www.pwc.com/us/en/health-industries/publications/health-care-social-media.jhtml. Accessed June 1, 2012.
  • 16.Tompson T, Benz J, Willcoxon N, et al. Finding Quality Doctors: How Americans Evaluate Provider Quality in the United States: Research Highlights. 2014 http://www.apnorc.org/projects/Pages/HTML%20Reports/finding-quality-doctors.aspx. Accessed 8/6/2014.
  • 17.2008 Update on Consumers’ Views of Patient Safety and Quality Information. 2008 http://www.kff.org/kaiserpolls/upload/7819.pdf.
  • 18.Greaves F, Millett C, Nuki P. England’s Experience Incorporating “Anecdotal” Reports From Consumers Into Their National Reporting System: Lessons for the United States of What to Do or Not to Do? Med Care Res Rev. 2014 May 16; doi: 10.1177/1077558714535470. [DOI] [PubMed] [Google Scholar]
  • 19.Patient Centered Outcomes Research Institute Mission and Vision. 2014 http://www.pcori.org/about-us/mission-and-vision/. Accessed September 10, 2014.
  • 20.King RB, Carlson CE, Shade-Zeldow Y, Bares KK, Roth EJ, Heinemann AW. Transition to home care after stroke: Depression, physical health, and adaptive processes in support persons. Research in Nursing & Health. 2001;24(4):307–323. doi: 10.1002/nur.1032. [DOI] [PubMed] [Google Scholar]
  • 21.Kripalani S, Jackson AT, Schnipper JL, Coleman EA. Promoting effective transitions of care at hospital discharge: A review of key issues for hospitalists. Journal of Hospital Medicine. 2007;2(5):314–323. doi: 10.1002/jhm.228. [DOI] [PubMed] [Google Scholar]
  • 22.Bardach NS, Asteria-Penaloza R, Boscardin WJ, Dudley R Adams. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ quality & safety. 2012 Nov 23; doi: 10.1136/bmjqs-2012-001360. 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Greaves F, Pape UJ, King D, et al. Associations between internet-based patient ratings and conventional surveys of patient experience in the English NHS: an observational study. BMJ quality & safety. 2012 Apr 20; doi: 10.1136/bmjqs-2012-000906. [DOI] [PubMed] [Google Scholar]
  • 24.Detz A, Lopez A, Sarkar U. Long-term doctor-patient relationships: patient perspective from online reviews. J Med Internet Res. 2013;15(7):e131. doi: 10.2196/jmir.2552. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Lopez A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012 Jun;27(6):685–692. doi: 10.1007/s11606-011-1958-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Kilaru AS, Meisel ZF, Paciotti B, et al. What do patients say about emergency departments in online reviews? A qualitative study. BMJ quality & safety. 2015 Jul 24; doi: 10.1136/bmjqs-2015-004035. [DOI] [PubMed] [Google Scholar]
  • 27.Greaves F, Laverty AA, Cano DR, et al. Tweets about hospital quality: a mixed methods study. BMJ quality & safety. 2014 Oct;23(10):838–846. doi: 10.1136/bmjqs-2014-002875. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Lagu T, Goff SL, Hannon NS, Shatz A, Lindenauer PK. A mixed-methods analysis of patient reviews of hospital care in England: implications for public reporting of health care quality data in the United States. Jt Comm J Qual Patient Saf. 2013 Jan;39(1):7–15. doi: 10.1016/s1553-7250(13)39003-5. [DOI] [PubMed] [Google Scholar]
  • 29.Sofaer S, Crofton C, Goldstein E, Hoy E, Crabb J. What Do Consumers Want to Know about the Quality of Care in Hospitals? Health Services Research. 2005;40(6p2):2018–2036. doi: 10.1111/j.1475-6773.2005.00473.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Goldstein E, Farquhar M, Crofton C, Darby C, Garfinkel S. Measuring Hospital Care from the Patients’ Perspective: An Overview of the CAHPS® Hospital Survey Development Process. Health Services Research. 2005;40(6p2):1977–1995. doi: 10.1111/j.1475-6773.2005.00477.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Braun V, Clarke V. Using Thematic Analysis in Psychology. Qualitative Research in Psychology. 2006;3(2):77–101. [Google Scholar]
  • 32.Introduction to the HCUP Nationwide Inpatient Sample (NIS) 2010. 2012 www.hcup-us.ahrq.gov/db/nation/nis/NISIntroduction2010.pdf Accessed 01/10/201.
  • 33.Bardach N, Asteria-Penaloza R, Goldman LE, Lyndon A, Dudley RA. From the Closest Observers of Patient Care: Consumer Understandings of Hospital Quality. AcademyHealth Research Meeting; Baltimore, MD. 2013. [Google Scholar]
  • 34.Who’s Online: Internet User Demographics: Pew Research Center’s Internet and American Life Project. 2011 May; http://pewinternet.org/Trend-Data/Whos-Online.aspx. Accessed January 29, 2012.
  • 35.Fox S, Duggan M. Health Online 2013. 2013 Jan 15; http://www.pewinternet.org/2013/01/15/health-online-2013/. Accessed March 13, 2013.
  • 36.Maxwell JA. A realist approach for qualitative research. Sage; 2012. [Google Scholar]
  • 37.Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007 Aug;42(4):1758–1772. doi: 10.1111/j.1475-6773.2006.00684.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Godlee F. Peer Review in Health Sciences. 2nd. London, UK: BMJ Books; 2003. pp. 219–235. [Google Scholar]
  • 39.Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. 192007 doi: 10.1093/intqhc/mzm042. [DOI] [PubMed] [Google Scholar]
  • 40.O’Malley AJ, Zaslavsky AM, Elliott MN, Zaborski L, Cleary PD. Case-mix adjustment of the CAHPS Hospital Survey. Health Serv Res. 2005 Dec;40(6 Pt 2):2162–2181. doi: 10.1111/j.1475-6773.2005.00470.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Pear R. New System for Patients to Report Medical Mistakes. 2012 http://www.nytimes.com/2012/09/23/health/new-system-for-patients-to-report-medical-mistakes.html?_r=0. Accessed September 22, 2012.
  • 42.Schlesinger M, Grob R, Shaller D, et al. Taking Patients’ Narratives about Clinicians from Anecdote to Science. N Engl J Med. 2015 Aug 13;373(7):675–679. doi: 10.1056/NEJMsb1502361. [DOI] [PubMed] [Google Scholar]
  • 43.Elliott MN, Zaslavsky AM, Goldstein E, et al. Effects of Survey Mode, Patient Mix, and Nonresponse on CAHPS® Hospital Survey Scores. Health Services Research. 2009;44(2p1):501–518. doi: 10.1111/j.1475-6773.2008.00914.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Giordano LA, Elliott MN, Goldstein E, Lehrman WG, Spencer PA. Development, implementation, and public reporting of the HCAHPS survey. Med Care Res Rev. 2010 Feb;67(1):27–37. doi: 10.1177/1077558709341065. [DOI] [PubMed] [Google Scholar]
  • 45.Elliott MN, Lehrman WG, Goldstein E, Hambarsoomian K, Beckett MK, Giordano LA. Do hospitals rank differently on HCAHPS for different patient subgroups? Med Care Res Rev. 2010 Feb;67(1):56–73. doi: 10.1177/1077558709339066. [DOI] [PubMed] [Google Scholar]
  • 46.Arah OA, ten Asbroek AH, Delnoij DM, et al. Psychometric properties of the Dutch version of the Hospital-level Consumer Assessment of Health Plans Survey instrument. Health Serv Res. 2006 Feb;41(1):284–301. doi: 10.1111/j.1475-6773.2005.00462.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Rosenthal E. As Hospital Prices Soar, a Stitch Tops $500. New York Times. Dec 3;:2013. [Google Scholar]
  • 48.Yegian JM, Dardess P, Shannon M, Carman KL. Engaged Patients Will Need Comparative Physician-Level Quality Data And Information About Their Out-Of-Pocket Costs. Health Affairs. 2013 Feb 1;32(2):328–337. doi: 10.1377/hlthaff.2012.1077. 2013. [DOI] [PubMed] [Google Scholar]
  • 49.Grob R, Schlesinger M, Davis S, Cohen D, Lapps J. The Affordable Care Act’s Plan For Consumer Assistance With Insurance Moves States Forward But Remains A Work In Progress. Health Affairs. 2013 Feb 1;32(2):347–356. doi: 10.1377/hlthaff.2012.1090. 2013. [DOI] [PubMed] [Google Scholar]
  • 50.Cost Sharing for Health Care: France, Germany, and Switzerland. 2009 Jan; https://kaiserfamilyfoundation.files.wordpress.com/2013/01/7852.pdf. Accessed May 20, 2015.
  • 51.Marmot MG, Smith GD, Stansfeld S, et al. Health inequalities among British civil servants: the Whitehall II study. Lancet. 1991 Jun 8;337(8754):1387–1393. doi: 10.1016/0140-6736(91)93068-k. [DOI] [PubMed] [Google Scholar]
  • 52.Rothman AA, Park H, Hays RD, Edwards C, Dudley RA. Can additional patient experience items improve the reliability of and add new domains to the CAHPS hospital survey? Health Serv Res. 2008 Dec;43(6):2201–2222. doi: 10.1111/j.1475-6773.2008.00867.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004 Aug;39(4 Pt 1):1005–1026. doi: 10.1111/j.1475-6773.2004.00269.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Hibbard JH, Greene J, Overton V. Patients with lower activation associated with higher costs; delivery systems should know their patients’ ‘scores’. Health Aff (Millwood) 2013 Feb;32(2):216–222. doi: 10.1377/hlthaff.2012.1064. [DOI] [PubMed] [Google Scholar]
  • 55.Greene J, Hibbard JH. Why does patient activation matter? An examination of the relationships between patient activation and health-related outcomes. J Gen Intern Med. 2012 May;27(5):520–526. doi: 10.1007/s11606-011-1931-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Hibbard JH, Greene J. What the evidence shows about patient activation: better health outcomes and care experiences; fewer data on costs. Health Aff (Millwood) 2013 Feb;32(2):207–214. doi: 10.1377/hlthaff.2012.1061. [DOI] [PubMed] [Google Scholar]
  • 57.Mitchell SE, Gardiner PM, Sadikova E, et al. Patient activation and 30-day post-discharge hospital utilization. J Gen Intern Med. 2014 Feb;29(2):349–355. doi: 10.1007/s11606-013-2647-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Hibbard JH, Cunningham PJ. How Engaged Are Consumers in Their Health and Health Care, and Why Does It Matter. Washington, D.C.: Center for Studying Health System Change; Oct, 2008. [PubMed] [Google Scholar]
  • 59.Kuehn BM. Patient safety still lagging: advocates call for national patient safety monitoring board. JAMA. 2014 Sep 3;312(9):879–880. doi: 10.1001/jama.2014.10310. [DOI] [PubMed] [Google Scholar]
  • 60.More Than 1,000 Preventable Deaths a Day Is Too Many: The Need to Improve Patient Safety. Health, Education, Labor, and Pensions: Subcommittee on Primary Health and Aging. 2014 [Google Scholar]
  • 61.Clancy CM. Patient Safety: One Decade after To Err Is Human. 2009 doi: 10.1097/PTS.0b013e3181c2114a. http://www.psqh.com/septemberoctober-2009/234-september-october-2009-ahrq.html. Accessed October 29, 2013. [DOI] [PubMed]
  • 62.Glickman SW. AcademyHealth Research Meeting. San Diego, CA: 2014. A Patient Reported System to Identify Medical Errors and Improve Safety in the Emergency Department. [Google Scholar]
  • 63.Jones S. Generations Online in 2009. 2009 http://pewresearch.org/pubs/1093/generations-online. Accessed January 29, 2012.
  • 64.Bardach N, Hibbard JH, Dudley RA. An aggregate analysis of 16 online public reporting Web sites and users’ and experts’ suggestions for improvement. Rockville, MD: 2011. Users of Public Reports of Hospital Quality: Who, What, Why, and How? [Google Scholar]
  • 65.Sofaer S, Crofton C, Goldstein E, Hoy E, Crabb J. What do consumers want to know about the quality of care in hospitals? Health Serv Res. 2005 Dec;40(6 Pt 2):2018–2036. doi: 10.1111/j.1475-6773.2005.00473.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Goldstein E, Farquhar M, Crofton C, Darby C, Garfinkel S. Measuring hospital care from the patients’ perspective: an overview of the CAHPS Hospital Survey development process. Health Serv Res. 2005 Dec;40(6 Pt 2):1977–1995. doi: 10.1111/j.1475-6773.2005.00477.x. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix
NIHMS844152-supplement-Appendix.docx (28.2KB, docx)

RESOURCES