Analyzing Online Reviews of Substance Use Disorder Treatment Facilities in the USA Using Machine Learning

INTRODUCTION

Substance use disorder (SUD) continues to have alarming impacts on morbidity and mortality.¹ Quality metrics for SUD remain undefined and less is known about the patient experience when accessing and obtaining care.² The search for treatment often begins online where individuals research facilities and are presented with aggregated online reviews. Online reviews are not validated or representative, yet their volume and use are expanding. Research into online reviews of healthcare presents an opportunity to identify trends in organic, patient-driven content.³ Informing the development of SUD treatment metrics with patient perspectives is important. We analyzed online reviews of US SUD treatment facilities to identify potential drivers of patient experience.

METHODS

We matched SUD treatment facilities from the Substance Abuse and Mental Health Services Administration (SAMHSA) to corresponding facilities on Yelp (review website). Facility records were matched using the shortest string matching Levenshtein distance, a linguistic method comparing name and address across SAMHSA and Yelp. Matched pairs were scored according the metaphone algorithm, a measure of phonetic similarity. The matches were manually reviewed and selected for analysis. We excluded non-SUD facilities (e.g., dentists), and similar to prior research, we selected facilities with at least five reviews to maximize the potential for identifying themes.⁴ We extracted the relative frequency of words and phrases (consisting of two or three consecutive words). We generated 25 latent Dirichlet allocation (LDA) topics using the MALLET implementation.⁵ LDA uses a dimension reduction procedure to identify latent topics in large quantities of text. Topics represent a list of words which cluster together. The distribution of LDA topics were extracted for each facility. Themes were categorized by independent research team review.

Ordinary least squares regression was used to identify topics associated with facility ratings. Effect size was measured using Pearson’s r. We identified 10 reviews each with the highest topic prevalence and assigned themes. We used Benjamini-Hochberg p-correction and p < 0.05 for indicating meaningful correlations.

RESULTS

Of 13,926 SAMHSA designated SUD facilities, we identified 12,938 online reviews matched to 1857 facilities over 15 years (09/2005–07/2020) and 533 facilities with >5 reviews (8149 reviews). The mean rating was 2.86 (1.0 SD). Review language correlated with positive or negative ratings is displayed in Figure 1.

Figure 1.

Language associated with positive and negative online ratings of substance use disorders. font size represents stronger correlation with positive or negative review and increased frequency of word usage is represented by darker shading.

We identified 16 total themes associated with positive and negative reviews (Table 1). The top five themes most correlated with a positive review included the following: long-term recovery (r = 0.66), dedicated staff (r = 0.63), dedication to patients (r = 0.56), group therapy experience (r = 0.43), and inpatient rehabilitation (r = 0.37). The top five themes most correlated with a negative review included the following: professionalism (r = −0.53), phone communication (r = −0.49), overall communication (r = −0.42), wait times in facility (r = −0.40), and management (r = −0.34).

Table 1.

Statistical Insights on Differential LDA Topics Across High and Low Facility Ratings. Significance Was Measured Using Paired, Two-Tailed t Test with Benjamini-Hochberg p-correction (p < 0.05)

Theme	Pearson r (95% CI) (facility ratings)	Top words	Example reviews (rephrased to preserve anonymity)
Long-term recovery	0.66 (.61, .71)	life, recovery, sober, program, years, saved, love, helped, grateful, addiction, amazing, hope, today, living, support	“[facility name] gave me the tools to get my life back and to learn how to live the life I was always meant to live. today i have a life beyond my wildest dreams. I am thankful and grateful to everyone on the hope staff for the excellent guidance and direction that I received. go in with an open mind and a willing heart”
Dedicated staff	0.63 (.58, .68)	staff, great, amazing, helpful, recommend, experience, feel, friendly, caring, professional, comfortable, made, dr, kind, highly	“very friendly and welcoming staff who go out of their way to make you feel comfortable. the counselors are knowledgeable and professional and the scheduling is very organized.”
Dedication to patients	0.56 (.50, .62)	treatment, recovery, clients, center, program, addiction, recommend, client, highly, centers, loved, professional, caring, facilities, support	“[facility name] sets the gold standard for treatment centers by embracing the whole person affected with addiction and their family. addiction is a mental illness, not a personal choice...their tailored treatment is combined with genuine concern and authentic caring…”
Group therapy experiences	0.42 (.35, .49)	therapy, group, experience, therapist, feel, sessions, felt, issues, therapists, found, things, individual, groups, process, skills	“I had a wonderful experience...i truly benefited from the process groups, recovery assignments... i also made numerous life-long friends!”
Inpatient rehab	0.37 (.30, .44)	program, house, detox, sober, days, stay, rehab, meetings, alcohol, living, week, 12, drugs, step, aa	“i came to [facility name] for medically supervised detox …the staff is kind, courteous, and attentive... they have regular counseling sessions, relapse prevention, and one on one therapy... i highly recommend [facility name] to anyone wanting to beat addiction.”
Family member	0.25 (.12. .33)	son, years, drug, passages, addiction, addict, drugs, son’s, abuse, year, husband, problem, alcohol, rehab, bridge	“addiction comes in many forms. my son’s was prescription drugs. [facility name] was a life saver...”
Care for loved one	0.18 (.10, .27)	family, facility, member, loved, patient, reviews, members, review, brother, negative, case, positive, read, friends, communication	“if you or a family member has trauma and/or substance abuse issues this is a top notch rehabilitation facility. I would recommend it to any family member, friend or client.”
Counseling services	0.16 (.08, .24)	services, classes, counselor, class, counseling, court, la, location, highly, dui, high, assessment, deal, helping, counselors	“the best, most biblical centered christian counseling i have ever experienced. they have improved myself and marriage.”
Children or minors	−0.26 (−.34, −.18)	daughter, child, kids, home, children, parents, send, talk, kid, school, admitted, unit, worse, girls, safe	“my daughters assigned social worker was in trading for the 1st week she was at [facility name]. i called many times trying to get feed back but nobody called me back. they discharged her without us having a family session.
Reception experience	−0.30 (−.37, −.22)	front, desk, woman, waiting, lady, man, walk, walked, door, rude, dont, open, attitude, guys, face	“slow service, disrespectful registration...nurse joking with security and flirting instead of doing her job.
Medications	−0.32 (−.39, −.24)	medication, meds, doctor, days, dr, medications, put, prescribed, dose, drug, sierra, prescription, taking, psychiatrist, methadone	“doctors did not listen to my pleas when i said specific medicines were not working for me and they simply just kept giving me alternative medicines and it had led to worsening my insomnia and anxiety would not recommend this place”
Insurance and billing	−0.33 (−.40, −.25)	insurance, money, pay, bill, paid, billing, company, charge, paying, cost, months, business, charged, card, payment	“they are overstretched. it is almost impossible to get an appointment within two weeks. the billing is all screwed up. for the longest time i was told that no copay was due (although according to my insurance contract there is a $20 copay for each visit). then all of a sudden i owed $300. my insurance has been billed for appointments that i did not attend and many visits do not show up on the insurance billing.”
Management	−0.34 (−.42, −.27)	patient, patients, medical, state, director, due, lack, information, complaint, management, poor, fact, practice, records, multiple	“one of the worst operated medical facilities that i have ever encountered. this company employs undereducated and incompetent staff that do not adhere to patient/client confidentiality clauses. engages in borderline criminal acts by supplying patient information to unauthorized individuals. this company has no remorse nor guilt towards contradicting official court orders...”
Wait times in office	−0.40 (−.47, −.33)	appointment, time, wait, doctor, office, minutes, waiting, hour, dr, appointments, front, hours, late, scheduled, schedule	“in the lobby 1 hour and 20 minutes and still waiting... just a reminder of why i stopped coming here.”
Communication	−0.42 (−.49, −.35)	told, asked, did not, day, called, made, wanted, needed, hours, wasn’t, left, leave, finally, home, felt	“this place is a complete joke, i went there this morning to get help and this doctor decided to dose me tomorrow morning because my pupils were too small...and did not even want hear what i was going through. this place is very unprofessional specially the doctor. this never happened to me where they told me to wait another 24 hours in order to get dosed”
Phone communication	−0.49 (−.55, −.42)	call, phone, called, back, calls, left, times, answer, person, hold, number, calling, information, speak, office	“they have answering problems! people at the desk 24/7 but they can never answer the phone, i can be on hold for 20 minutes and then they hang up the phone, I am just trying to book a quick appointment why is it such a hassle?”
Professionalism	−0.53 (−.59, −.47)	place, patients, people, do not, rude, worst, give, horrible, money, unprofessional, joke, absolutely, terrible, awful, treat	“nasty staff who treat patients like garbage. place should be closed.” “terrible. very loud and disorganized. very strange place, no one seems to be in charge here.”

DISCUSSION

Using machine learning, we investigated narrative content from over 8000 online reviews of US SUD treatment facilities. Research in online reviews has grown to identify potential indicators of patient experience within hospitals, emergency departments, urgent care centers, and nursing facilities.⁶ This study builds upon prior work to provide a national analysis of reviews pertaining to SUD treatment.⁴ Consistent with prior research, we focused on facilities with ≥5 reviews in order to maximize the potential for identifying themes across facilities and avoid instances of facilities with infrequent or rare reviews. Future research will need to focus on identifying a threshold number of reviews per facility to provide more local actionable insights.

Themes within positive reviews reveal insights on topics which are difficult to quantify in structured surveys and highlight interpersonal connections such as a focus on long-term recovery or the dedication of staff. These potential drivers of positive reviews can be used to emphasize a facility’s core mission and support broader guiding principles for SUD care. Themes associated with negative reviews emphasize a lack of professionalism and provide actionable patient-driven observations to guide improvements in communication and waiting times. This study demonstrates the ability to analyze a collective “digital patient voice” and provides a patient-centered approach toward informing quality measures for SUD treatment.

Limitations include retrospective design, selection, and responder biases inherent to online reviews. Online platforms use proprietary software to filter inappropriate, invalid, or inaccurate reviews and do not verify the identity of the individual posting a rating or reviewer. Nonetheless, online platforms provide an unstructured, organic, and accessible venue for patients to share experiences and inform healthcare.

Funding

Funding provided by NIH NIDA 1R21DA050761.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.