Abstract
Objectives:
This study aims to evaluate the utility of the Global Appraisal of Individual Needs (GAIN) Recommendation and Referral Report (GRRS) as guided by American Psychiatric Association diagnosis criteria and American Society of Addiction Medicine (ASAM) guidelines for treatment planning and placement.
Methods:
GAIN data were gathered between March 2018 and June 2020 from a total of 82 agencies and 245 clinicians as part of a program evaluation of agencies receiving public funding through the Mid-State Health Network under contract with the Michigan Department of Health and Human Services and the Office of Recovery Oriented Systems of Care. Of the 1,395 patients aged 18 and older, 1,027 GRRS reports were produced by clinical staff. Kappa and Rho analyses were used to measure rates of clinician agreement with the recommendations produced by the GRRS report based on patient interviews.
Results:
Clinicians agreed with the GRRS preliminary diagnostic recommendations 88-100% of the time, with Kappa scores indicating excellent agreement by ranging from .6 to .9. For an average patient, 41 out of 46 treatment planning statements generated by the GRRS were used by clinicians, with moderate to high correlation was indicated by Rho scores ranging from .62-.82. The percent agreement for all ASAM dimension ratings was greater than 99%, with kappa scores of .98 and higher.
Conclusions:
This study demonstrates the utility and efficiency of the GRRS as a clinical decision support system to support diagnosis, treatment, and placement in routine practice.
Keywords: Substance Use Disorder (SUD), Clinician, Self-Report, Assessment
Introduction
Patient placement criteria developed by the American Society of Addiction Medicine (ASAM) was constructed to efficiently employ services by simplifying treatment admission processes.1 However, the criteria do not provide specific instruments to obtain patient information or algorithms for translating them into ratings.1,2 Consequently, there is variation in how data is collected and interpreted by clinicians, presenting a need to integrate the criteria within existing scales that are broadly circulated.
The Global Appraisal of Individual Needs (GAIN) instrument is a series of assessment measures designed to evaluate substance use based on patient interviews.3 As of July 2022, the GAIN has been used to collect over 1.1 million assessments by 20,390 staff in 5,504 agencies from all of the states in the U.S., provinces of Canada and 11 other countries. GAIN data has been used to generate 869 publications and detailed norms and psychometrics have been provided on 306,608 individuals. Scales and Subscales of the GAIN have been validated in diverse patient populations.4–7 The GAIN Recommendation and Referral Report (GRRS) is the clinical report generated from data collected by the GAIN that was established in collaboration with clinicians and researchers from over 3 dozen systems of care to translate patient reports from the semi-structured interview into preliminary diagnostic impressions, treatment planning recommendations, and level of care placement guided by the ASAM Criteria.8 This analysis examines the utility of the GRRS in routine clinical practice.
Methods
Data Source
Data were collected March 2018- June 2020 from 82 agencies, covering all 10 of Michigan’s Prepaid Inpatient Health Plans (PIHPs). The PIHP represents the full continuum of care of publicly funded treatment available in Michigan. Clinicians (n=245) trained and certified by the GAIN Coordinating Center on the Administration of the GAIN-I Core and were professionally licensed to make Diagnosis by the State of Michigan. Data were collected under general consent to treatment and de-identified at patient and program levels under the 42 CFR part 2 exception for program evaluation and research.
Survey Measures
Patient-reported symptoms were collected using the GAIN, comprised of the following subsections: 1) Background and Treatment Arrangements; 2) Substance Use (Alcohol, Marijuana and Other Drugs); 3) Physical Health; 4) Risk Behaviors and Disease Prevention; 5) Mental and Emotional Health; 6) Environment and Living Situation; 7) Legal (Civil and Criminal); and 8) Vocational (School, Work, Financial). The GAIN was selected after reviewing multiple measures supporting ASAM because of its established norms, psychometrics, scoring, reporting tools, and workforce development protocols. Primary SUDs as defined by the GAIN include cannabis, alcohol, opioids, stimulants, and other drugs.
The GRRS is the principal clinical interpretive report generated to translate findings from the GAIN into preliminary diagnostic impressions, treatment planning statements, and ratings on each of the 6 ASAM dimensions for level of care placement. Problems are rated on a dimension of none, past, current moderate or severe in the primary dimension. A secondary dimension reports the recency of treatment. The report flags potential inconsistencies and decision points for clinicians. Staff have the ability to directly edit reports as well as agree/disagree with the report findings. All staff were trained, received coaching based on digital recording and were certified under the GAIN’s mastery approach to workforce development.9
Participants
Data were collected from 1,395 patients (ages 18 and older) through the GAIN; GRRS reports were generated and edited by clinical staff from 1,027 of these assessments. The majority of the sample was Caucasian (60%), male (51%), and between 26 to 49 years old (70%). The majority of patients (61%) reported symptoms consistent with 5 or more clinical problems including SUDs (84%), victimization (73%), tobacco use disorder (72%), depression (56%), physical health problems (52%), illegal activity (47%), trauma (47%), and violence (43%).
Statistical Analysis
Kappa was used to identify rates of clinician agreement with the results of the GRRS reported diagnosis and treatment planning statements by adjusting for the prevalence rate of responses. The kappa metric adjusts the percentage of agreement based on how rare an occurrence is from 0 to 1.0. Rank order correlation for ASAM Placement was used to compare the computer-generated ratings based on patient report with final clinician ratings. Rho agreement ranged from 0-none to 1-agreement.
Results
Table 1 demonstrates levels of agreement between preliminary diagnosis from the GRRS (based on patient reports) and final clinician diagnosis. Clinicians agreed with the GRRS impressions 88%-100% of the time, deleted 12%, added 10% of their own input, and identified misrepresentation of information in less than 3% of patients. Kappa scores ranged between .6 to .9, indicating excellent agreement between clinicians and GAIN results. Table 2 demonstrates agreement between preliminary treatment planning recommendations from the GRRS and the statements retained by clinicians in their final narrative and treatment plan. Of the 47,613 original treatment planning statements across the sample and six ASAM dimensions, 40,759 (86%) were found to be consistent with patient-reports on the GAIN. An additional 6,629 (14%) were dropped, (not reviewed/signed off by a licensed clinician), 225 (<1%) were edited and 775 (2%) new treatment planning statements were added. On average, clinicians utilized 41 out of 46 (88%) of the treatment planning statements. Rho scores ranged between .62-.82. indicating moderate to high correlation. Appendix Table A.1 demonstrates the agreement between ASAM dimension ratings recommended by the GAIN based on patient self-report with the final rating from the clinicians in their biopsychosocial. This rating is based on problem severity by whether or not they are currently in treatment/intervention to make the 7 cells across the Table 1. Since ratings are based on self-report, there is also a category for an illogical response that the clinician will have to probe (e.g., treatment with no history of problems). The percent agreement for all categories was greater than 99%, with kappa scores of .98 and higher indicating excellent agreement.
Table 1.
Agreement between preliminary diagnosis from the GRRS and clinician’s final diagnosis
| Past Year Psychiatric Diagnosis | Self Report | Clinician Impression | % Agreement | Kappa |
|---|---|---|---|---|
| Any substance use disorder | 84% | 89% | 95% | 0.79 |
| Alcohol use disorder | 45% | 47% | 97% | 0.94 |
| Cannabis use disorder | 25% | 30% | 95% | 0.86 |
| Opioid use disorder | 25% | 25% | 99% | 0.98 |
| Stimulant use disorder | 34% | 35% | 99% | 0.97 |
| Other use disorder | 8% | 7% | 99% | 0.90 |
| Any mental health disorder | 63% | 61% | 94% | 0.88 |
| Any Internalizing disorder | 54% | 53% | 96% | 0.91 |
| Mood disorder | 45% | 40% | 92% | 0.85 |
| Depressive disorder | 44% | 39% | 92% | 0.85 |
| Anxiety disorder | 39% | 32% | 92% | 0.83 |
| Trauma disorder | 25% | 24% | 93% | 0.82 |
| Suicide problems | 22% | 21% | 100% | 0.99 |
| Any Externalizing disorder | 46% | 38% | 91% | 0.82 |
| Attention deficit/hyperactivity Disorder | 44% | 36% | 92% | 0.83 |
| Conduct disorder | 30% | 18% | 88% | 0.66 |
| Any Personality Disorder | 6% | 5% | 96% | 0.63 |
Table 2.
Agreement between the number of preliminary treatment planning recommendations from the GRRS and the final number of statements used by clinicians
| ASAM Dimension | Preliminary number of statements from GRRS based on patient reports | Final number of treatment planning statements used by Clinicians | Within person correlation (Rho) treatment planning statements based on patient report vs used by clinicians |
|---|---|---|---|
| B1. Intoxication and Withdrawal | 3.43 | 2.96 | 0.71 |
| B2. Bio - Medical | 8.98 | 7.42 | 0.62 |
| B3. Psych - Behavioral | 8.79 | 7.95 | 0.82 |
| B4. Readiness for Change | 6.66 | 6.00 | 0.82 |
| B5. Relapse Potential | 10.15 | 9.09 | 0.75 |
| B6. Recovery Environment | 8.34 | 7.25 | 0.70 |
| Total number of statements | 46.36 | 40.66 | 0.68 |
Discussion
Clinicians exhibited trust in the accuracy of the GRRS recommendations for diagnosis, treatment placement and ASAM ratings when based on patient report measures on the GAIN. Evaluative models that provide accurate and clear diagnostic impressions are necessary to guide treatment planning for SUDs.10 Treatment planning statements and placement recommendations focused on specific needs of patients produce more positive treatment outcomes than methods providing less individualized treatment plans.11,12 When matched with proper care, patients experience improved treatment retention, lower drug use and addiction severity, and higher readiness for a lower level of care placement faster than those that did not receive coordinated services.12 An effective allocation of resources also maximizes patient outcomes and is imperative in areas of low access to substance use treatment services.10 By translating the GAIN into recommendations that are for the most part used, the GRRS also demonstrates it efficiency as a clinical decision support system.
Limitations of this study include the lack of GRRS reports from all GAIN assessments, as well as a restriction to the state of Michigan. Future qualitative data from clinicians would be beneficial in identifying implementation barriers/challenges and usefulness of the feedback tool.
Overall, this study demonstrates the utility and efficiency of the GRRS as a clinical decision support system for diagnosis, treatment planning and ASAM placement. This is also a key advance for furthering the move towards linking assessment to services provided in order to individualize care.
Acknowledgements
This presentation would not have been possible without the support of the state, Community Mental Health Association of Michigan (CMHAM), the Prepaid Inpatient Health Plan (PHIP) regions, programs, and line staff as well as the patients who completed the assessment. The authors also thank CMHAM / PHIP network staff Angela Smith-Butterwick, Heather Rosales, Cori Miles, Sandra McGuiness, and Barbara Estrada for comments on the original draft. The opinions expressed here are those of the authors and do not represent positions of the programs, state or others.
Sources of Support:
Funding was provided by Chestnut Health Systems, the Community Mental Health Association of Michigan, and by the National Institute on Drug Abuse [R21 DA046738]. The opinions expressed here are those of the authors and do not represent positions of the programs, state or others.
Declaration of Interest:
Dr. Welsh has received consulting fees received from Applied Clinical Intelligence LLC (ACI Clinical). Ms. Sitar and Dr. Dennis have no disclosures to report.
Appendix Table A.1.
Agreement between the preliminary placement ratings from GRRS (columns) with the final clinician rating by ASAM Dimensions
| ASAM Dimension | 0. no problem/any Tx history | 1. no problem/no Tx history | 2. past probs/past Tx history | 3. current mod sev problems/no current Tx | 4. Current high sev problems/no current Tx | 5. lifetime problems/current Tx | 6. current mod sev problems/current Tx | 7. Current high sev problems/current Tx | % Agreement | Wieghted Kappa |
|---|---|---|---|---|---|---|---|---|---|---|
| B1- Intoxication & Withdrawal | 3% | 39% | 38% | 4% | 9% | 5% | 0.3% | 2% | 99.5% | 0.99 |
| B2- Biomedical | 13% | 3% | 27% | 5% | 2% | 23% | 13.8% | 13% | 99.6% | 0.99 |
| B3- Psychological/Behavioral | 0% | 5% | 17% | 16% | 20% | 7% | 8.8% | 26% | 99.8% | 1.00 |
| B4- Readiness for Change | 0% | 1% | 18% | 40% | 10% | 8% | 18.8% | 3% | 99.7% | 1.00 |
| B5- Relapse Potential | 1% | 0% | 1% | 2% | 14% | 14% | 20.5% | 47% | 99.1% | 0.98 |
| B6- Recovery Environment | 1% | 1% | 2% | 33% | 41% | 0% | 8.9% | 13% | 99.6% | 1.00 |
Footnotes
Conflicts of Interest: None.
References
- 1.Mee-Lee D, Shulman GD, Fisherman M, Gastfriend DR, Miller. The ASAM Criteria: Treatment Criteria for Addictive, Substance-Related, and Co-Occurring Conditions, 3rd Edition. 3rd ed. The Change Companies; 2013. [Google Scholar]
- 2.Padwa H, Mark TL, Wondimu B. What’s in an “ASAM-based Assessment?” Variations in Assessment and Level of Care Determination in Systems Required to Use ASAM Patient Placement Criteria. J Addict Med. 2022;16(1):18–26. doi: 10.1097/ADM.0000000000000804 [DOI] [PubMed] [Google Scholar]
- 3.Dennis ML, Titus JC, White M, Unsicker J, Hodgkins D. Global appraisal of individual needs: Administration guide for the GAIN and related measures. Published online 2003. Accessed February 5, 2017. https://www.researchgate.net/profile/Michael_Dennis2/publication/267383950_GAIN_Global_Appraisal_of_Individual_Needs_Administration_Guide_for_the_GAIN_and_Related_Measures/links/54838a6f0cf25dbd59eb0ef0.pdf
- 4.Conrad Kendon J, Conrad KM, Mazza J, Riley BB, Stein Mark B, Dennis ML. Dimensionality, hierarchical structure, age generalizability, and criterion validity of the GAIN’s Behavioral Complexity Scale. Psychol Assess. 2012;24(4):913–924. doi: 10.1037/a0028196. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.McDonell Michael G, Comtois Katherine Anne, Voss William D, Morgan Amanda H, Ries Richard K. Global Appraisal of Individual Needs Short Screener (GSS): psychometric properties and performance as a screening measure in adolescents. Am J Drug Alcohol Abuse. 2009;35(3):157–160. doi: 10.1080/00952990902825421 [DOI] [PubMed] [Google Scholar]
- 6.Coleman-Cowger Victoria H., Dennis Michael L., Funk Rodney R., Lennox Richard D.. Comparison of the Addiction Severity Index (ASI) and the Global Appraisal of Individual Needs (GAIN) in Predicting the Effectiveness of Drug Treatment Programs for Pregnant and Postpartum Women. J Subst Abuse Treat. 2013;44:34–41. doi: 10.1016/j.jsat.2012.02.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Dennis ML, Funk R, Godley SH, Godley MD, Waldron H. Cross-validation of the alcohol and cannabis use measures in the Global Appraisal of Individual Needs (GAIN) and Timeline Followback (TLFB; Form 90) among adolescents in substance abuse treatment. Addict Abingdon Engl. 2004;99 Suppl 2:120–128. doi: 10.1111/j.1360-0443.2004.00859.x [DOI] [PubMed] [Google Scholar]
- 8.Dennis Michael L., Smith Corey N., Funk Rodney R., Olsen Scott D.. Global Appraisal of Individual Needs (GAIN) Program Feedback to MI Recovery Oriented Systems of Care (ROSC) Project Network and Providers. Powerpoint presented at: August 28, 2020. [Google Scholar]
- 9.Titus JC, Smith DC, Dennis ML, Ives M, Twanow L, White MK. Impact of a training and certification program on the quality of interviewer-collected self-report assessment data. J Subst Abuse Treat. 2012;42(2):201–212. doi: 10.1016/j.jsat.2011.10.017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Ritter Alison, Mellor Richard, Chalmers Jenny, Sunderland Matthew, Lancaster Kari. Key Considerations in Planning for Substance Use Treatment: Estimating Treatment Need and Demand. J Stud Alcohol Drugs. 2019;18:22–30. doi: 10.15288/jsads.2019.s18.22 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Agniel Denis, Almirall Daniel, Burkhart Q, et al. Identifying optimal level-of-care placement decisions for adolescent substance use treatment. Drug Alcohol Depend. 2020;212(1). doi: 10.1016/j.drugalcdep.2020.107991 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Stallvik Marianne, Gastfriend David R., Nordahl Hans M.. Matching patients with substance use disorder to optimal level of care with the ASAM Criteria software. J Subst Use. 2014;20(6):389–398. doi: 10.3109/14659891.2014.934305 [DOI] [Google Scholar]