Rate Your Addiction Treatment Facility: Exploring the Quality Chasm Online

The opioid crisis has shed light on components of the addiction treatment system in the USA that could benefit from improved quality. Despite initiatives to define, measure, and enhance addiction treatment services and outcomes that predate the 2001 Institute of Medicine’s “Quality Chasm” report¹ on the importance of quality in healthcare systems, there remains stark variation in treatment services and outcomes across programs. Importantly, because addiction treatment services have traditionally been provided through a system that runs in parallel to “mainstream” medicine and are often overseen by non-medical staff, many patients, family members, and clinicians are unable to determine the quality of care provided in specialized drug treatment facilities (SDTFs).

There is increasing urgency around expanding access to and improving the quality of treatment of substance use disorders as deaths related to drug overdoses have soared and concerns about ineffective and predatory addiction services have risen.^{2, 3} However, echoing broader challenges in benchmarking quality in healthcare, there has not been consensus around what indicators are important to measure. Different stakeholders may have differing perspectives on what factors are important and how much to weigh them. Nonetheless, a patient-centered approach is essential to creating quality measures that are meaningful to and reflect the needs and values of people that may use them to inform major healthcare decisions.

Just as people frequently turn to the internet to share their experiences and opinions about traditional consumer goods, they also use the internet to share their experiences and opinions about health services. As such, it can be a source of rich qualitative data around what drives positive and negative experiences in healthcare. Rating platforms such as Yelp and Google Reviews aggregate this information via a five-star rating system and narrative comments. Insights from these data may be able to shed additional light on what is important to the patient experience. Studies of themes emerging from online reviews of hospital,^4–6 emergency department,⁷ urgent care,⁷ pharmacy,⁸ and nursing home⁹ settings have been conducted, but not much is known about the content of online reviews for substance use treatment facilities.

It is in this setting that Agarwal and colleagues conducted an analysis of 7635 online ratings from Yelp and Google Reviews from July 2010 to August 2018 for the 485 of Pennsylvania’s 536 SDTFs that were listed in the 2016 National Directory of Drug and Alcohol Abuse Treatment Facilities and had online reviews on either platform.¹⁰ The objectives of the study were to identify themes present within online ratings associated with positive and negative reviews and to compare these with elements measured by existing facility surveys. Using machine learning techniques for topic modelling, they identified themes associated with positive (5-star) and negative (1-star) narrative reviews. Themes most correlated with 5-star reviews related to focus on recovery (r = 0.53), experiencing a life-changing moment (r = 0.32), staff helpfulness (r = 0.43), compassionate care (r = 0.37), and professionalism (r = 0.29). On the other hand, themes most correlated with 1-star reviews highlighted concerns around wait time (r = 0.41), poor accommodations (r = 0.26), poor phone communication (r = 0.24), medications offered (r = 0.24), and appointment availability (r = 0.23).

For the second objective, the authors tested for correlation between the themes identified from online reviews and facility service categories from the 2017 National Survey of Substance Abuse Treatment Services (NSSATS) administered by the Substance Abuse and Mental Health Services Administration (SAMHSA). They found that themes identified from online reviews were represented in 3 of 14 SAMHSA facility-level categories and 12 of 162 SAMHSA service codes. Additionally, they correlated NSSATS responses with 5-star and 1-star reviews and found that outpatient medication-based treatment (r = 0.15), self-pay treatment facilities (r = 0.12), and lesbian, gay, bisexual, or transgender clients (r = 0.12) were most associated with 5-star reviews; psychiatric emergency walk-in services (r = − 0.17), state-funded treatment facilities (r = − 0.16), and integrated primary care service (r = − 0.15) were most associated with 1-star reviews.

The authors’ approach leverages the vast amount of narrative information would not be captured by traditional methods of gathering rich qualitative data for this purpose, such as in-depth interviews, focus groups, and surveys—which are also more resource intensive. Unprompted, organically generated data may reveal patient-centered themes that could potentially be masked by prompted or structured assessments. Additionally, the use of a machine learning technique facilitates processing of large volumes of data in a dynamic fashion.

However, there are limitations to using this method. Firstly, though review platforms claim to have proprietary means of assessing authenticity, data fidelity remains a concern in this type of analysis—we are all familiar with the concept of fake reviews. Not all people who rate or write reviews are verified to have directly received a service from a facility. Moreover, only a small proportion of people who utilize a service will go on to review it, and they may not be a representative sample of all people who have used the service. As shown in the study, the distribution of star ratings is bimodal in the extremes, and differences exist in the mean ratings that do and do not have accompanying narrative text. And, since substance use disorders are stigmatized and many review platforms have a non-anonymous, semi-social design, the review rate and content may be further biased. In fact, though there were 7635 reviews, the mean number of reviews per facility was 15.3 (SD 24.0, median 7, IQR 3–6) over the course of eight years—or, on average, approximately two reviews per facility per year. This functional “digital divide” likely biases results further. Since data was obtained for one state, additional research would be needed to confirm whether themes generated and associated with positive and negative experiences in one geographic region remain similar in other areas. Lastly, the definition of SDTFs used combines widely heterogeneous types of treatment programs (i.e., outpatient, residential, inpatient), providing a variety of treatment services (i.e., withdrawal management, pharmacotherapy, behavioral therapies), for a range of disparate conditions (i.e., alcohol, opioid, stimulant, marijuana use disorders) and excludes the growing component of the addiction treatment system that includes office-based and primary care–based addiction treatment. Overlooking this heterogeneity ignores the nuances of how different facility types, services, and recommended treatments for specific substance use disorders may influence patient expectations and experiences. The weak correlations between extreme ratings and the SAMHSA service codes is likely a reflection of this heterogeneity.

Nonetheless, this study adds to a growing body of literature around using ambient digital data to draw insights into the healthcare experience. The themes associated with positive and negative experiences can supplement information from other sources to create a more holistic view of factors that are important to patients and their families. Several other studies of online review data have identified similar factors as associated with positive (good staff behaviors, compassion) and negative experiences (wait time, poor communication, poor accommodations)^4–9; the endurance of these findings across settings highlights systemic issues around the design and provision of healthcare services. The discrepancy between themes identified in online reviews and information in the NSSATS is not surprising, however, since NSSATS and review sites have different objectives. NSSATS is primarily used to create a descriptive catalog of treatment resources; on the other hand, review sites are designed to capture experiential data to crowdsource information about facilities’ services and quality.

Facilities that treat substance use disorders need meaningful implementation of quality measurement. Predatory practices take advantage of this lack of consistent regulation or quality measurement and have harmed—and continue to harm—people seeking help for addiction. Particularly online, in the absence of other reliable quality indicators, people may lean heavily on information gleaned from review sites and other digital sources as their only means of evaluating a service and may be at increased risk for misdirection and patient brokering.² Creating quality measures that take into consideration patients’ experiences and other outcomes and implementing these measures in ways that are easily seen and understood are key to protecting patients and families from misinformation and harm. In the future, regulatory, quality, and digital agencies can work together to facilitate dissemination and utilization of these indicators for enhanced patient safety.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they do not have a conflict of interest.