Online Reviews of Mental Health Treatment Facilities: Narrative themes associated with positive and negative ratings

Abstract

Objectives:

Previous studies indicate that patient satisfaction with mental healthcare is correlated with both treatment outcomes and quality of life. This study aimed to 1) describe online reviews of mental health treatment facilities, including key themes in review content, and 2) evaluate the correlation between narrative review themes, facility characteristics, and review ratings.

Methods:

United States National Mental Health Services Survey (N-MHSS) facilities were linked to their corresponding Yelp pages, created between March, 2007 and September, 2019. Correlations between review ratings and both machine-learning generated latent Dirichlet allocation topics and N-MHSS reported facility characteristics were measured using Spearman’s rank correlation coefficient, and significance was defined by a Bonferroni adjusted p<.001.

Results:

Of 10,191 unique mental health treatment facilities, 1,383 (13.6%) had relevant Yelp pages with 8,133 corresponding reviews. The number of newly reviewed facilities and the number of new reviews increased throughout the study period. Narrative topics positively correlated with review ratings included caring staff (Spearman’s rho=.39) and non-pharmacologic treatment (rho=.16). Topics negatively correlated with review ratings included rude staff (rho=−.14) and safety and abuse (rho=−.13). Of 126 N-MHSS survey items, 11 were positively correlated with review rating, including “outpatient mental health facility” (rho=.13), and 33 were negatively correlated with review rating, including accepting Medicare (rho=−.21).

Conclusions:

Narrative topics provide information beyond what is currently collected through the N-MHSS. Topics associated with positive and negative reviews, such as staff attitude towards patients, can guide improvement in patients’ satisfaction and engagement with mental healthcare.

In the United States (US), approximately 1 in 5 people will experience a mental illness each year, and less than 50% will receive treatment.(1) While most evidence points to increasing provision of treatment over time, studies also show an increasing mental health burden both nationally and internationally.(2–4) Factors contributing to the treatment gap include underfunding - in 2015, the mental health burden in the US was 2.7 times greater than the proportion of health funds allocated to mental health(3) - but also patients’ engagement in their own care. Patient engagement impacts treatment retention and therapeutic outcomes.(5) While engagement depends on patients’ perceptions of the quality of care they receive, few studies have analyzed factors associated with positive or negative experiences of mental health treatment.(6, 7) Patient satisfaction is associated with fewer psychiatric symptoms and better quality of life.(8)

The implementation of mental healthcare quality measures lags behind that of other fields of medical care.(9) The majority of the quality measures that have been endorsed by the US National Quality Forum and used in major quality reporting programs, such as the Center for Medicare and Medicaid Services’ Hospital Compare, are related to screening and assessment rather than patient-centered outcomes.(10) According to the Patient-Centered Outcomes Research Institute, end-users of research, including patients, should be involved in defining outcomes that are “meaningful and important to patients and caregivers.”(11) Psychiatric hospitals included in Hospital Compare and evaluated through the Inpatient Psychiatric Facility Quality Reporting Program do not report star ratings, and patients who “received psychiatric or rehabilitative services” are excluded from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) because the “instrument is not designed to address…the behavioral health issues pertinent to psychiatric patients”.(12, 13) The Substance Abuse and Mental Health Services Administration (SAMHSA) collects data on mental health treatment facilities through the National Mental Health Services Survey (N-MHSS), but the survey does not include quality or patient satisfaction measures.(14)

Online review sites, such as Yelp, Google, and Facebook, provide platforms for word-of-mouth communication about healthcare, including mental health services.(15–23) Risks with using online review data include fraudulent reviews and over-representation of extreme opinions. Reviews do, however, offer advantages: they allow for narrative reporting, are public, and are available to both patients and their support networks.(15, 24) Studies have demonstrated that review ratings and narrative themes tend to correlate with both existing national surveys, such as HCAHPS and the National Survey of Substance Abuse Treatment Services, and in some cases, with health outcomes.(16, 21, 25) Review themes also provide insight into drivers of satisfaction, as measured by review ratings, that are not captured by standard surveys.(16, 18, 19, 21, 22, 25)

In order to identify factors associated with patients’ or their families’ positive or negative experiences with mental healthcare, we analyzed Yelp reviews of mental health facilities registered in SAMHSA’s Treatment Locator. As of March 31, 2020, Yelp had 35 million average monthly mobile app users and 211 million reviews.(26) Given Yelp’s extensive usership and the lack of standardized patient quality and satisfaction measures in US mental healthcare, patients may turn to the site when selecting or evaluating a mental health facility. Understanding the factors that influence user reviews of facilities could inform areas of focus for reducing unmet mental health needs in the US. The purpose of this analysis was threefold: to understand the extent to which Yelp is used to review mental health treatment facilities, to describe narrative themes in Yelp reviews of mental health treatment, and to identify narrative themes and facility services correlated with review ratings.

Methods

This study was considered exempt by the University of Pennsylvania Institutional Review Board. The Yelp data used in the study represents an academic dataset generated directly by Yelp for research and includes all US facilities tagged as “health” facilities according to Yelp’s developer documentation.(27) It is updated daily, and extends beyond content available through the Yelp application programming interface. SAMHSA facility data and N-MHSS data are freely available for download from the SAMHSA website.

Study sample

To evaluate online reviews for a validated group of facilities, we sought to match SAMHSA mental health facilities to their corresponding Yelp pages. This matching afforded the added advantage of allowing for comparison between Yelp review ratings and the N-MHSS data associated with each SAMHSA facility.

We downloaded all SAMHSA data and Yelp data in September, 2019. Facilities were added to Yelp between March, 2007 and September, 2019. Facility data included name; location; and reviews, including both review star rating (one to five) and narrative review content.

From SAMHSA’s Treatment Locator, we identified 10,191 unique “mental health” facilities, last updated between February, 2017 and September, 2019. Facilities are eligible for registration in the Treatment Locator if they are funded by a state mental health agency, administered by the US Department of Veterans Affairs, or licensed by a state agency or national organization to provide mental health treatment. Eligible facilities are identified through surveys of state mental health authorities and medical organizations. All included facilities complete the N-MHSS, and according to the 2018 N-MHSS, the “survey universe” of mental health treatment facilities was 14,159. (14)

Each SAMHSA facility was matched to a corresponding Yelp page according to a probabilistic string-matching algorithm with a sensitivity of 88.9% and a specificity of 95.1%.(28, 29) The list of SAMHSA mental health facilities successfully matched to Yelp pages included medical centers with psychiatric units. In the majority of such cases, the corresponding Yelp page was for the parent organization, and most of the reviews pertained to general medical care rather than mental healthcare. We used a term search strategy to remove facilities with reviews using high proportions of words more likely to refer to general medical care.²⁰ This step was validated through a combination of hand-coding and topic modeling. Further details are available online.

Review analysis

To understand Yelp usage over time, we calculated the number of new reviews and the number of facilities receiving a first review by yearly quarter. To evaluate review content, we used a machine learning based natural language processing model, Latent Dirichlet allocation (LDA). LDA operates under the assumption that documents, in this case Yelp reviews, can be described by a pre-specified number of topics. Topics are defined by distributions of commonly co-occurring words, and each review is then defined by a distribution of topics. A topic most strongly defined by the words “staff”, “helpful”, and “friendly” might then be highly representative of a review such as, “Great place!! Front desk staff were very helpful.” Authors DCS and HM independently reviewed the ten words and reviews most strongly associated with each of the 30 LDA topics and, where review content was consistent, assigned the topics a theme. Discrepancies were adjudicated by a third reviewer (RMM). This process resulted in identification of 13 consistent topics. This approach, and the ratio of meaningful topics to all LDA topics, was consistent with prior Yelp studies. (16, 19, 21)

Survey analysis

Patient ratings of mental health treatment facilities may be correlated with existing objective facility metrics, as has proven true in hospitals, substance use treatment facilities, and nursing homes.(16, 21, 25) In mental healthcare, for instance, evidence from qualitative analyses suggests patients have concerns specific to pharmacologic and inpatient treatment.(30, 31) The N-MHSS is an annual survey conducted by SAMHSA and is the only national survey of both public and private mental health treatment facilities.(14) The survey includes 18 service categories divided into 126 binary service codes, and it covers such information as facility type, special programming, and availability of emergency mental health services. In order to identify how narrative analysis of review content may support or augment data currently collected by SAMHSA, we sought to measure the correlation between N-MHSS survey items and Yelp review ratings.

Statistical analysis

To compare star ratings with both N-MHSS services and LDA topics, we used Spearman’s rank correlation coefficient, which is suited to comparisons of nonparametric data. Star ratings are ordinal, ranging from 1 to 5, while LDA probability distributions are generally non-normally distributed continuous variables between 0 and 1. N-MHSS services are binary variables. Both the correlations between N-MHSS services and star ratings and between star ratings and LDA probabilities were performed at the level of the Yelp review. Significance was defined by a Bonferroni-corrected p<.001, corresponding to a corrected alpha of .0000072.(32) All statistical analyses were performed using R, version 3.6.1.(33)

Results

Study sample

Of the 10,191 unique SAMHSA facilities, 2,403 (23.6%) were matched to Yelp “health” facilities. Of these matches, 1,383 (13.6%) remained following filtering by general medical terms (flow diagram available online). As summarized in Table 1, the final sample differed meaningfully from SAMHSA mental health facilities without dedicated Yelp pages by geographic distribution, with over-representation of facilities located in the West, and on several survey items: among service settings, “hospital inpatient” and “partial hospitalization/day treatment” were over-represented, as was special programming for specific groups, such as “young adults” and “veterans”.

Table 1.

Baseline characteristics of SAMHSA mental health facilities on Yelp compared to all SAMHSA mental health facilities^a

Characteristic	SAMHSA mental health facilities (N=10,191)				p
	With Yelp reviews (N=1,383)		Without Yelp reviews (N=8,808)
	N	%	N	%
Region					<.001
Northeast	254	18.4	1941	22.2
South	358	25.9	2682	30.7
Midwest	303	21.9	2303	26.4
West	468	33.8	1806	20.7
Service settingsb
Hospital inpatient	396	28.6	1233	14.0	<.001
Outpatient	1113	80.5	7012	79.6	.478
Partial hospitalization/day treatment	369	26.7	7544	14.4	<.001
Residential	218	15.8	1237	14.0	.097
Telemedicine/telehealth	513	37.1	3358	38.1	.481
Offers special programming for people pertaining to the following groupsb
Young adults	356	25.7	1907	21.7	<.001
Seniors	460	33.3	2300	26.1	<.001
Veterans	291	21.0	1436	16.3	<.001
LGBT	426	30.8	1838	20.9	<.001
Co-occurring mental and substance use disorders	827	59.8	4430	50.3	<.001
Living with HIV/AIDS	204	14.8	920	10.4	<.001
Experienced trauma	681	49.2	3844	43.6	<.001
Children/adolescents with serious emotional disturbances	492	35.6	3268	37.1	.287
Living with serious mental illness	721	52.1	4327	49.1	.040
Offers emergency mental health servicesb
Crisis intervention team	643	46.5	4481	50.9	.003
Psychiatric emergency walk-in services	512	37.0	2914	33.1	.004

^aProportions were compared by chi square tests.

^bCategories not mutually exclusive, and differences therefore tested individually.

The facilities had a total of 8,133 reviews, corresponding to a mean number of 5.88 reviews per facility (range 1 to 107, median 3). The mean word count per review was 170.2±164.8. The distribution of review ratings was bimodal, with 57.5% one-star (lowest possible) and 26.5% five-star (highest possible) reviews. As shown in Figure 1, both the number of reviews and the number of facilities receiving a first review increased from 2006 to 2019.

Figure 1.

Number of new mental health facility reviews and mental health facilities receiving a first review on Yelp by quarter between January, 2006 and June, 2019

Review analysis

Of the 13 LDA topics, four were positively correlated with review ratings, seven were negatively correlated with review ratings, and two were not significantly correlated with review ratings, according to a Bonferroni corrected p<.001. The topics, redacted sample reviews, and correlations with review ratings are presented in Table 2. The strength of correlations was generally greater for positive topics, e.g. “helpful and friendly staff” (Spearman’s rho = .39), than for negative topics, e.g. “rude ancillary staff” (rho = −.14).

Table 2.

Yelp review topics with sample redacted reviews and Spearman’s rank correlation to review rating

Online review theme	Example reviewa	Spearman’s rho
Topics positively correlated with review rating
Caring staff	…everyone here are so nice and so helpful…from the front staff all the way to the doctors!…Im very happy with the service I have received and continue to receive. Thank you everyone….	.39
Life-saving treatment	If you are ready to get better and you work the program you will succeed! Staff is caring and compassionate…They will fight for you which at times you will hate but in the end, it will save your life.	.28
Non-pharmacologic treatment modalities	I did outpatient individual therapy, dietitian support…and…group therapy all in one building…Every…provider I worked with really genuinely cared for me.	.16
Therapeutic alliance with primary provider	Dr M is the best. He listens and he shows concern as well as empathy…I would highly recommend Dr M.	.11
Topics negatively correlated with review rating
Rude ancillary staff	For how amazing and incredible the residential staff is, the front desk staff is arrogant, rude, and comes across as not caring. Front desk is the first impression… its a shame its being ruined by your front desk staff	−.14
Safety and abuse	I left with more psychological problems than when I arrived. The staff are not trained…I laughed when they tried to stop bullying since the staff were the ones bullying children with name calling and brutal physical assaults…	−.14
Billing and insurance	I was lied to about insurance coverage…told multiple times that my insurance would cover it all and then got billed…they'll only take your insurance if you work for certain companies.	−.11
Scheduling	My family member has called almost every week to schedule an appointment now that she did her intake, and each time she calls they tell her that she has not been assigned a counselor and they transfer her to…voicemail. None of her messages have been returned… It shouldnt take 3 months to get an appointment…every time we ask to speak to the director, we are told that they are not in.	−.09
Treatment for specific diagnosis	…Brought a person here who needed help with drug dependency and depression…patient was treated as if they were a criminal being booked into a jail. Still in search of dignified and compassionate care!	−.08
Communication with family	No communication with the families. No support for the families. When we picked up our loved one we were given ZERO information. No info about options of what the next steps should be once the patient leaves.	−.08
Pharmacotherapy	My counselor was great…but the Doctor got me hooked on antidepressants and after 4 years tried to discharge me after I missed some counseling appts. I ended up…thanking my counselor and told her that I would not be back and that I would ween myself off the meds.	−.06
Topics non-significantly correlated with review rating b
Child and adolescent treatment	I was 14 years old when I was at this place. I hated the way the staff would treat us kids, they are there to help us not destroy our happiness…Continuously put residents down as if they were trash. Im almost 30 now and I'm still scarred and scared of that place.	.01
Inpatient facility amenities	Their food is horrible…Beds are trash. Worst "hospital" ever. Extremely limited food, outside activity, phone calls, visitation, recreational exercise. The nurses go in your room every 5 seconds slamming doors open and shut in your shared room.	−.05

^aElements of the reviews have been redacted in order to preserve user anonymity

^bCorrelation considered significant where p below Bonferroni corrected alpha of .001 (p<.(001/139)=7.2×10⁻⁶)

Reviews most represented by the generally positive topic “nonpharmacologic treatment modalities” (rho=.16) included mentions of “individual therapy”, “Yoga”, “dietitian support”, “group therapy”, “dialectical behavior skills”, “family workshops”, and “trauma program[s]”. Those for the generally negative topic “safety and abuse” (rho=−.14) mentioned specific concerns about “name calling” and being “violated…verbally”, “physical assault” by both patients and staff, “sexual assault”, “theft”, and “neglect”. Of the seven topics negatively correlated with review rating, two were related to clerical services: “billing and insurance” (rho=−.11) and “scheduling” (rho=−.09).

Survey analysis

Facilities offered on average 40±11 of the 126 N-MHSS services. In total, 11 services were significantly correlated with five-star reviews, and 33 were significantly correlated with one-star reviews. Table 3 shows the 10 services with strongest positive and negative correlations to Yelp review ratings. Among facility types, “outpatient mental health facility”(rho=.13) and “residential treatment center for adults” (rho=.07) were positively correlated with review rating, while “psychiatric hospital” (rho=−.19) was negatively correlated with review rating.

Table 3.

SAMHSA services most positively and negatively correlated with Yelp review ratings, according to Spearman’s rank correlation coefficient

Services	Spearman’s rho
Services most positively correlated with review rating
Facility Type
Outpatient mental health facility	.13
Residential treatment center (RTC) for adults	.07
Facility Operation (e.g. Prviate, Public)
Veterans Affairs Medical Center	.05
Exclusive Services
Serves Veterans only	.05
Service Settings
Outpatient	.06
Special Programs/Groups Offered
Lesbian, gay, bisexual, or transgender (LGBT) clients	.10
Persons with post-traumatic stress disorder (PTSD)	.06
Ancillary Services
Housing services	.07
Intensive case management	.06
Psychosocial rehabilitation services	.05
Services most negatively correlated with review rating
Payment/Insurance/Funding Accepted
Medicare	−.21
Medicaid	−.18
Military insurance	−.16
State welfare or child and family services funds	−.12
Facility Type
Psychiatric hospital or psychiatric unit of a general hospital	−.19
Service Settings
Hospital inpatient	−.18
Emergency Mental Health Services
Psychiatric emergency walk-in services	−.17
Language Services
Services for the deaf and hard of hearing	−.16
Treatment Approaches
Psychotropic medication	−.15
Tobacco/Screening Services
Screening for tobacco use	−.13

^aAll p-values below Bonferroni-corrected alpha of. 001 (p<.(001/139)=7.2×10⁻⁶)

As with the LDA topic “non-pharmacologic treatment modalities” (rho=.16), N-MHSS codes for special programs and for ancillary services were generally positively correlated with review rating, e.g. programming for “lesbian, gay, bisexual or transgender clients” (rho=.10) and “housing services” (rho=.07). On the other hand, as with the LDA topic “billing and insurance”, N-MHSS codes for specific forms of payment and insurance, especially public insurance, were generally negatively correlated with review rating, e.g. “Medicare” (rho=−.21), as were both the LDA topic “pharmacotherapy” (rho=−.06) and the N-MHSS code “psychotropic medication (rho=−.15).

Discussion

This study has three main findings. First, individuals are increasingly reviewing mental health facilities online. The percent of SAMHSA facilities identified on Yelp (14%) was between the percent of hospitals (31%) and skilled nursing facilities (11%) identified on the site in prior studies.(16, 25) The number of Yelp pages and reviews of mental health facilities is expected to continue to increase: in the four months between June and September of 2019, 806 Yelp pages were added under the site’s “counseling and mental health” tag. Of note, facilities in the West and facilities offering inpatient treatment were over-represented on Yelp. Yelp is California-based, which may explain the former, and large facilities may be both more likely to offer inpatient services and more likely to have Yelp pages. The bimodal distribution of online review ratings, with high proportions of one-star and five-star reviews, was consistent with distributions observed for other health facilities.(16, 19) In the absence of formal outlets for patients to report their experiences of mental healthcare, we expect the volume of patients choosing to do so through more informal means, such as online reviews, to continue to grow.

Second, correlations between review ratings, narrative topics, and N-MHSS services were consistent with previous findings in both qualitative analyses of mental healthcare and analyses of online reviews. An analysis of online reviews of substance use treatment facilities in Pennsylvania identified the positive review theme “life-changing experiences” and the negative review theme “medication needs”, consistent with themes we identified as correlated positively (“life-saving treatment”) and negatively (“pharmacotherapy”) with online reviews of mental health facilities.(21) The strength of correlations reported in the substance use treatment paper between themes and review ratings were slightly larger than those reported here, perhaps due to a combination of fewer LDA topics and binary treatment of review ratings. Studies of inpatient psychiatric care point to patient concerns around safety, restrictions on freedom, communication, and stigma. (34, 35) In our analysis, there were more negative comments associated with inpatient services than with outpatient services, and reviews with poor ratings were more likely to mention issues of “safety and abuse”, “communication with family”, and “rude ancillary staff”. Themes associated with more positive ratings, like “relationship with primary provider” and “caring staff”, also correspond to existing evidence: a study of 61 inpatients found that “feeling cared for” and “positive qualities of staff” were themes common to positive patient appraisals of care.(34) In general, evidence suggests that patient “satisfaction” is perhaps most dependent on staff-patient communication.(36) Communication with staff has been consistently identified as a theme in online reviews of healthcare, including in reviews for emergency departments, hospitals, and substance use treatment facilities.(16, 18, 21) Among mental healthcare facility reviews, we found that topics related to communication, namely “caring staff” and “rude ancillary staff”, were those most strongly correlated with positive and negative review ratings, respectively.

The third main finding is that narrative topics identified in reviews include actionable items that may improve satisfaction with mental healthcare. Satisfaction is inherently important - how patients feel about the care they receive should matter to providers - but in mental healthcare in particular, it may also be associated with fewer psychiatric symptoms and better quality of life.(8) Evidence regarding the relationship between satisfaction and physical health outcomes is mixed, but generally supports positive correlations between patient evaluations of communication with both doctors and nurses and more objective measures of quality.(25, 36, 37) Many negative reviews of mental healthcare facilities mentioned “rudeness” explicitly. While “rudeness” is not unique to mental healthcare, it may contribute to patients’ internalized stigma, which is associated with decreased help-seeking and disempowerment among those with mental illness.(38, 39) Mental healthcare facilities could use the information from Yelp reviews to guide patient-centered interventions. For instance, negative reviews consistently mentioning rude staff could prompt clinic managers to organize targeted communication and anti-stigma training, with the potential to improve patient engagement in mental healthcare.

Limitations

Limitations inherent to online reviews include that reviews skew towards extremes, and that fake reviews may escape algorithmic detection. In addition, Yelp users are not representative of those needing mental healthcare. According to Yelp, US users are almost equally distributed between the ages of 18–34, 35–54, and 55 and over. Users skew heavily towards at least some college (82%), and half of users have an income above $100,000 a year.(26) According to a 2018 SAMHSA report, “any mental illness” was most common among those aged 18 to 49, and among those 18 and older, approximately 64% had at least some college education and 24% were living below the poverty line.(1) Yelp is therefore at least over-representative of those with high education and income. Yelp notably does not report on the racial-ethnic demographics of its users, and several racial-ethnic groups in the US, particularly Black and Latinx people, face persistent barriers to mental healthcare.(41) In the future, dialogue with those with lived experience of mental health treatment, and those with specific barriers to care, will be necessary to determine how well Yelp themes encompass factors contributing to patient satisfaction.

This study was further limited by a relatively low proportion of SAMHSA facilities with Yelp pages. Google and Facebook reviews are not accessible through application programming interfaces, but may include facilities not found on Yelp..(21) Sub-analyses by facility type, e.g. inpatient vs. outpatient and adult vs. child, were not conducted due to limitations in sample size. These analyses, and evaluations of emergency psychiatric care and of tele-psych communication, will be important as further online review data becomes available.

A limitation of natural language processing is that identified themes cannot always be easily addressed.(23) For instance, despite a negative correlation between pharmacologic treatment and review ratings, in many cases evidence supports the use of medication in treating mental illness. Knowledge of patient aversion to medication, however, can help set the foundation for therapeutic alliance.

Finally, while we were able to compare facility ratings to standard facility-level characteristics through N-MHSS data, there is no nationally reported facility rating system for mental health facilities in the United States (e.g. Hospital Compare for hospitals), against which to measure Yelp review ratings. Evidence from studies where such comparisons were possible point to correlations ranging from 0.09 to 0.5.(16, 20)

Conclusion

Online reviews are powerful in that they are unscripted, and as such, the themes that arise from these review narratives are naturally “patient-centered”. Our study’s findings regarding review ratings and the narrative content of reviews are consistent with both national survey data and the available literature. In the absence of the wide-spread adoption of validated patient-centered outcome measures in mental health, online reviews of mental health facilities have the potential to guide interventions to improve patient satisfaction.

Highlights.

• This study utilizes natural language processing of 8,133 online patient reviews, in combination with review ratings and National Mental Health Services Survey data, to understand thematic features associated with positive and negative ratings of mental health care

• Themes associated with positive ratings included caring staff, non-pharmacologic treatment, and therapeutic alliance, while those associated with negative ratings included concerns around safety and abuse, pharmacotherapy, and poor communication with family

• Limitations include that online reviews skew towards extreme opinions, and that the only available national-level data for comparison is survey data that does not include patient ratings, perspectives, or outcomes