An Evaluation of Evidence Underpinning Management Recommendations in Tobacco Use Disorder Clinical Practice Guidelines

Abstract

Introduction

Clinical practice guidelines(CPGs) are important tools for medical decision-making. Given the high prevalence and financial burden associated with tobacco use disorder(TUD), it is critical that recommendations within CPGs are based on robust evidence. Systematic reviews(SRs) are considered the highest level of evidence, thus, we evaluated the quality of SRs underpinning CPG recommendations for TUD.

Methods

We used PubMed to search for CPGs relating to TUD published between January 1, 2010 and May 21, 2021. SRs were extracted from CPG references and evaluated using Preferred Reporting Items for Systematic Reviews and Meta-Analyses(PRISMA) and A MeaSurement Tool to Assess Systematic Reviews(AMSTAR-2) tools. We then compared SRs conducted by the Cochrane Collaboration with non-Cochrane SRs using a Mann-Whitney U test and determined associations between PRISMA and AMSTAR-2 extracted characteristics using multiple regression.

Results

Our search generated 10 CPGs with 98 SRs extracted. Mean PRISMA completion was 74.7%(SD = 16.7) and mean AMSTAR-2 completion was 53.8%(SD = 22.0) across all guidelines. Cochrane SRs were more complete than non-Cochrane studies in the PRISMA and AMSTAR-2 assessments. The regression model showed a statistically significant association between PRISMA completion and AMSTAR-2 rating, with those classified as “low” or “moderate” quality having higher PRISMA completion than those with “critically low” ratings.

Conclusion

We found substandard adherence to PRISMA and AMSTAR-2 checklists across SRs cited in TUD CPGs. A lack of recent SRs in CPGs could lead to outdated recommendations. Therefore, frequent guideline updates with recently published evidence may ensure more accurate clinical recommendations and improve patient care.

Implications

Systematic reviews used to underpin clinical practice guideline recommendations influence treatment decisions and, ultimately, patient outcomes. We found that many systematic reviews underpinning tobacco use disorder guideline recommendations were out of date and unsatisfactory in reporting and quality. Thus, including newer systematic reviews containing more recently conducted trials and better reporting could alter recommendations and improve the rate of successful tobacco cessation attempts.

Introduction

Tobacco use disorder (TUD) is the leading cause of preventable morbidity and mortality in the United States.¹ This pervasive public health issue is problematic, but also serves as a significant financial burden. According to the Center for Disease Control and Prevention, smoking-related illness accounts for more than US$300 billion each year, with upwards of US$170 billion stemming from direct medical costs.² Owing to the high prevalence and substantial financial burden attributable to the management of TUD and disease sequelae, it is essential that clinicians are informed on the most recent and methodologically robust evidence when counseling patients on available treatments.

Clinical practice guidelines (CPGs) assist clinicians and patients in making decisions regarding clinical circumstances. In 2008, the US Public Health Service updated the CPG, Treat Tobacco Use and Dependence Guidelines, which provides recommendations for clinicians, patients, and health policy makers regarding available options for treating TUD.³ Over the past several decades, there has been an increase in access to treatment options for TUD, further emphasizing the importance of the validity and methodological quality of the studies underpinning CPG recommendations.^3–5 For example, since the release of the TUD treatment guidelines, many states have expanded Medicaid coverage to include at least one guideline-recommended treatment option.^3–5 The Joint Commission has implemented accreditation requirements for hospitals and other institutions to deliver evidence-based TUD interventions for all patients with significant medical comorbidities.³ The US Military and Veteran’s Hospital Administration has also expanded coverage to include guideline-recommended treatments for TUD.⁶ Despite the widespread use of these CPGs in the management of TUD, questions regarding the methodological quality of supporting evidence upon which CPG recommendations are made pose significant problems for clinicians and patients alike.

Previous studies have demonstrated significant variability in the quality of systematic reviews (SRs) that serve as the evidentiary base upon which CPG recommendations are made.^7–9 For example, an appraisal of the American Psychiatric Association (APA) Practice Guideline for the Treatment of Patients with Schizophrenia found the majority of systematic reviews included within these guidelines were of moderate quality.⁸ Ross et al. also reported suboptimal methodological quality of SRs included in opioid use disorder CPGs, with common underperforming areas including: title labeling, risk of bias, and protocol registration.¹⁰ Because SRs serve a key role in formulating CPG recommendations—and ultimately have direct downstream effects on patient care—it is crucial that these SRs are of the utmost methodological quality to increase the validity of evidence supporting CPG recommendations. Therefore, our objectives will be to: (1) assess the overall methodological and reporting quality of SRs used to develop CPGs for the treatment of TUD; (2) determine whether variations in the quality of these SRs exists in TUD CPGs; and (3) evaluate for factors contributing to variability in these quality scores.

Methods

This cross-sectional study did not qualify as human subjects research and therefore did not require Institutional Review Board oversight. The study protocol—which includes all search strategies, inclusionary criteria, and methodology—was developed a priori. To facilitate reproducibility and transparency of study results, all study materials have been made available on Open Science Framework (osf.io/93k2z/).

Outcomes

Our primary objective was to determine the reporting and methodological quality of SRs cited in the CPG recommendations for the management of TUD. As secondary objectives, we: (1) determined the extent to which SRs conducted by a Cochrane group are cited in the CPG for TUD; and (2) evaluated differences in PRISMA (Preferred Reporting Instrument for Systematic Reviews and Meta-Analyses) and AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews 2) scores between Cochrane and non-Cochrane SRs.

Identifying CPGs

One of us (T.T.) conducted a PubMed search to identify CPGs relevant to the management of TUD. The search string used to perform this search was developed using a model from the Canadian Agencies for Drugs and Technologies in Health¹¹ The exact search string used is available as part of the online protocol on Open Science Framework. Search results were uploaded to Rayyan QRCI (https://rayyan.qcri.org) for title and abstract screening. Selection of pertinent CPGs was based on the definition provided by the Institute of Medicine.¹² To be included, each CPG must have: (1) been related to the management of TUD; (2) been published between January 1, 2010 and May 21, 2021; and (3) for feasibility, been available in the English language. Three investigators (S.S., R.M., and L.C.) screened records in triplicate and masked the others’ responses. T.T. was available for third-party adjudication, if necessary.

Identifying Systematic Reviews and Meta-Analyses

Following initial CPG screening, two investigators (R.M. and L.C.) screened the reference section for all CPGs meeting inclusion criteria. Investigators identified all SRs cited in the CPG. To qualify as a SR, the reference had to: (1) meet the definition of a SR––as defined by the PRISMA-Protocols¹³; (2) be available in the English language; and (3) be cited in at least one of the CPGs.

Data Extraction and Scoring

Three investigators (S.S., R.M., and L.C.) underwent training led by one of the senior authors (T.T.) prior to the commencement of data extraction. Following training, these investigators scored all SRs using a pilot-tested Google form. To ensure consistency and understanding between investigators, we first scored 5 systematic reviews using the PRISMA and AMSTAR-2 instruments. Investigators compared results with the trainer prior to continuing. After completing this calibration exercise, investigators scored the remaining SRs according to the instrument instructions^.14,15 In addition to scoring SRs using the PRISMA and AMSTAR-2 forms, the following study characteristics can be seen in Supplementary Table 1: (1) year of publication; (2) participant population; (3) intervention; (4) number of included primary studies; (5) sample size of all included primary studies; and (6) study design of included primary studies. As with screening, all responses were extracted in triplicate followed by a final consensus meeting to resolve discrepancies. T.T. served as a third-party adjudicator, if necessary.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

PRISMA—a 27-item checklist—was developed by an expert panel to improve the reporting of SRs. Each included SR was scored using the PRISMA checklist in accordance with the methodology from previous studies.^7–10 Scores were assigned according to whether the SR met (“yes” = 1 point), partially met (“partial yes” = 0.5 point) or did not meet (“no” = 0 point) criteria for each of the 27 items. Overall scores were calculated as a proportion of criteria met.

A MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2)

AMSTAR-2 consists of 16 items that appraise SRs synthesizing evidence from randomized and/or nonrandomized studies.¹⁵ AMSTAR-2 scoring was based on the methodology of previous studies.^7–10 Investigators assigned scores for each of the 16 AMSTAR-2 items according to the following: “yes” if the SR met all item criteria; “partial yes” if some but not all item criteria were met; and “no” if item criteria were not met. The same point scoring system used for the PRISMA checklist was used for the AMSTAR-2 checklist. Of note, AMSTAR-2 items 11, 12, and 15 are specific to SRs containing meta-analyses (MA). Therefore, those items will receive a “N/A” response if the SR lacked MA. Thus, SRs lacking MA will be scored out of a total 13 items rather than 16. Overall scores will be calculated as a proportion of criteria met. Final appraisal ratings of “high”, “moderate”, “low”, or “critically low” will be assigned in accordance with the AMSTAR-2 calculator.¹⁵ Lastly, AMSTAR-2 is specific for SRs regarding interventions; therefore, if a SR included in our sample is not specific to an intervention for TUD, that SR will be excluded from the AMSTAR-2 analysis but will be included in the PRISMA analysis.

Secondary Analysis

We evaluated whether Cochrane SRs had improved quality of reporting and methodology compared to non-Cochrane SRs. This decision to make this comparison was based upon previous evidence suggesting that the quality of reporting within Cochrane SRs is superior to that of non-Cochrane SRs.^16,17 Therefore, to investigate the difference in reporting quality of SRs underpinning treatment recommendations for TUD, the SRs in our sample were separated into one of two categories: (1) those conducted by a Cochrane group; and (2) those conducted by a non-Cochrane group. We compared percent adherence to the PRISMA and AMSTAR-2 checklist between Cochrane and non-Cochrane SRs.

Statistical Analysis

Descriptive statistics were used for PRISMA and AMSTAR-2 scores, and measures of central tendency were reported for overall percent completion and individual items. A multiple regression model was constructed to determine associations between PRISMA completion, AMSTAR appraisal, and other study characteristics. Lastly, a Mann-Whitney U test was used to compare PRISMA and AMSTAR-2 scores between cohorts of Cochrane and non-Cochrane SRs. Stata 16.1 (StataCorp, LLC, College Station, TX) was used to compute all analyses.

Results

Our search of PubMed returned 2205 records. Following title and abstract screening, 1987 records were excluded because they were not CPGs. Full-text analysis yielded 111 additional exclusions for being in a language other than English, 96 were not relevant to TUD, and one was the wrong publication type. Our final sample of 10 CPGs pertaining to TUD can be seen in Table 1. We identified 1259 unique references from these CPGs and ended up with a final sample of 98 (7.78%) SRs via the process seen in Figure 1.

Table 1.

Characteristics of the Included Clinical Practice Guidelines

Clinical practice guideline	Year of publication	Geographical region	References per guideline	Systematic reviews per guideline	Systematic reviews as a proportion of all studies cited by CPG
Guidelines for smoking cessation (JCS 2010)—digest version	2012	Tokyo, Japan	85	3	3.5%
South African tobacco smoking cessation clinical practice guideline	2013	Cape Town, South Africa	86	17	19.8%
EPA guidance on tobacco dependence and strategies for smoking cessation in people with mental illness	2014	Cambridge, England	293	12	4.1%
Behavioral and pharmacotherapy interventions for tobacco smoking cessation in adults, including pregnant women: US Preventive Services Task Force Recommendation Statement	2015	Philadelphia, PA, United States	47	13	27.7%
Smoking Cessation, Version 1.2016, NCCN Clinical Practice Guidelines in Oncology	2016	Huntington, NY, United States	260	44	16.9%
Recommendations on behavioural interventions for the prevention and treatment of cigarette smoking among school-aged children and youth	2017	Ottawa, ON, Canada	29	2	6.9%
Society for Perioperative Assessment and Quality Improvement (SPAQI) Consensus Statement on Perioperative Smoking Cessation	2020	Toronto, ON, Canada	118	23	19.5%
Primary care interventions for prevention and cessation of tobacco use in children and adolescents: US Preventive Services Task Force Recommendation Statement	2020	Chicago, IL, United States	54	1	1.9%
Initiating pharmacologic treatment in tobacco-dependent adults. An official American Thoracic Society Clinical Practice Guideline	2020	New York, NY, United States	185	7	3.8%
Interventions for tobacco smoking cessation in adults, including pregnant persons: US Preventive Services Task Force Recommendation Statement	2021	Chicago, IL, United States	102	16	15.7%
Total			1259	138	11.0%

Figure 1.

PRISMA flow diagram of screening and sample selection.

Inclusion and Study Characteristics

The 98 SRs obtained for our assessment were published between 1989 and 2021. Ninety-one SRs (92.86%) included adult participants, 5 (5.10%) included a mixed pool of adults and adolescent participants, and 2 (2.04%) examined studies with only adolescent participants. The most prevalent funding source identified was government (55/98; 56.12%). A conflict of interest statement was included in 82 SR (82/98; 83.67%). Eighty-one SRs (81/98; 82.65%) performed a meta-analysis. The interventions being assessed in these 98 SRs were drugs on 45 occasions (45/98; 45.92%), combinations of interventions on 12 occasions (12/98, 12.24%), devices on 3 occasions (3/98; 3.06%), and other for the remaining 38 SRs (38.78%).

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

All 98 SRs were appraised according to the PRISMA checklist. The mean PRISMA completion among included SRs was 74.7%, (SD = 16.68) with a range of 25.9–100%. Three of the 27 PRISMA questions—item 3 (rationale for the review), item 4 (explicit statement of study objectives), and item 21 (presentation of study results)—had a mean score of 0.95 or higher. PRISMA items receiving lower mean scores were item 1 (presence of the phrase SR or MA in the study title) with a mean score of 0.39 (SD = 0.41), and item 5 (reference to a study protocol) with a mean score of 0.30 (SD = 0.36). SRs published in the Cochrane Database of Systematic Reviews had higher PRISMA mean completion (M = 84.85, SD = 5.15) compared to non-Cochrane SRs (M = 66.4, SD = 18.22).

A MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2)

Of the 98 included SRs, two without TUD interventions were excluded from AMSTAR-2 assessment. Therefore, we assessed AMSTAR-2 adherence for the remaining 96 included SRs. The mean AMSTAR-2 percent completion among the eligible SRs was 53.77% (range 6.25–93.75%). AMSTAR-2 item 11a (whether an appropriate statistical combination of results was used) had the highest percent completion at 0.93, whereas the item with lowest adherence was item 3 (whether SR authors provided the rationale for their selection of the study designs included in the SR) with a percent completion of 0.14 (SD = 0.34). We found that the overall study appraisal assigned by the AMSTAR-2 instrument included 41 SRs (41/96; 42.7%) receiving a “critically low” rating, 43 SRs (43/96; 44.8%) receiving a “low” rating, and 12 SRs (12/96; 12.5%) receiving a “moderate” rating. None of the included SRs received a “high” AMSTAR-2 rating.

A multiple regression analysis was performed that modeled the combination of the publication year, intervention type, whether a COI statement was listed, design of included studies, AMSTAR-2 rating, and funding of the SR. The model was statistically significant (F_{(df = 11, 84)} = 7.72; p < .001) and accounted for 43.77% of the variance of PRISMA completion percentage. Within this adjusted model, studies with higher quality AMSTAR-2 ratings completed more PRISMA items on average; compared to studies with “critically low” ratings, those with “low” ratings were 15.19% (SE = 3.38; t = 4.49, p < .001) more complete and those with “moderate” ratings were 13.17% more complete (SE = 4.27; t = 3.08, p = .003). No other associations within the adjusted model were statistically significant.

Secondary Analysis

Of the 96 SRs included in the AMSTAR-2 appraisal, 43 (44.8%) were conducted by a Cochrane group. Cochrane SRs had a higher percentage of inclusion of AMSTAR-2 items compared to non-Cochrane SRs (71.9% vs. 39.1%; Mann Whitney U Test: z = –7.39, p < .0001). Of the 43 Cochrane SRs, 8 (74.4%) received “moderate” ratings, 32 received “low” ratings, and 3 received “critically low” ratings. Of the 53 non-Cochrane SRs, 4 (7.6%) received “moderate” ratings, 11 (21.8%) received “low” ratings, and 38 (71.7%) received “critically low” ratings.

Discussion

Our findings indicate PRISMA adherence among SRs cited as supporting evidence in CPGs for the management of TUD is suboptimal. By appraising SR methodological quality using the AMSTAR-2 instrument, we found that SRs receiving higher AMSTAR-2 ratings were more likely to adequately report PRISMA items compared to SRs receiving “critically low” AMSTAR-2 ratings. In addition, results from our secondary analysis suggest that both PRISMA and AMSTAR-2 scores were superior among SRs completed by the Cochrane group compared to those completed by non-Cochrane groups.

PRISMA Adherence and AMSTAR-2 Ratings of Study Characteristics

The majority of deficiencies in methodological reporting, as defined by PRISMA, were found to correlate with year of publication. SRs published after the development of PRISMA were associated with higher scoring. A study conducted by Doshi et al. had similar findings and recommended that practitioners look at literature on nicotine cessation cautiously until newer SRs with better documentation and reporting have been conducted.¹⁸ Reporting checklists, like the PRISMA and AMSTAR-2 assessments, are important to ensure quality reporting and consistency between studies, and can help writers of CPGs to make more informed recommendations. Moreover, studies published in the Cochrane Database of Systematic Reviews had consistently higher PRISMA and AMSTAR-2 scores for the majority of checklist items when compared to non-Cochrane SRs. Given the differing adherence and quality of Cochrane and non-Cochrane SRs, CPGs and clinicians may benefit from increased inclusion of recommendations underpinned by SRs published by the Cochrane group. Due to the quality of evidence and timeliness of publication, multiple Cochrane SRs have been published since these CPGs were established which could update or expand their content.^19–21

The absence of any SRs receiving an AMSTAR-2 scoring of “high” is concerning, but cannot be entirely attributed to inadequate methodology. Low scoring can be partially attributable to the strict measures of the recently updated AMSTAR-2 assessment and researchers having little time to adjust practices since its creation in 2017. To score as “high,” the study must have a nearly flawless assessment. A perfect score is difficult considering all items on the assessment have the same influence on scoring, but some are seemingly more dispensable than others. For example, the lowest overall completed item in the AMSTAR-2 assessment was item 3, and broadly asked if researchers provided a rationale for inclusion of specific study types used in their research. In theory, item 3 should not be difficult to incorporate into studies, but until recently many current standards of reporting have not been common practice.²² Omissions such as item 3 are affecting study ratings, but are of little relevance to an audience not assessing completeness of reporting. Since the update in 2017, researchers have had little time to adjust their writing to match that of the AMSTAR-2 assessment, consequently contributing to the low assessment scores. Only about 16% of our included SRs were published after 2017, so it is understandable that the percent adherence within our sample may be of lower quality. However, we deemed it appropriate to measure adherence with the current validated standard measure because results from these older SRs, in part, are still dictating the direction of clinical practice for patients with TUD. There have been many studies across various medical fields that show little to no “high” AMSTAR-2 scores with similar reasoning.^23,24 While others claim the original AMSTAR assessment showed much higher scoring of studies than the updated version.²⁵ However, it has been maintained that a thorough assessment of reporting quality is necessary, and while AMSTAR-2 is more stringent than the original AMSTAR, it is necessary to score with this instrument as it is the gold standard in which to abide.¹⁵ The supposed inadequacies of the AMSTAR-2 assessment could be enough to skew potential “high’‘ scores to “moderate.” However, the fact that nearly 90% of studies in our sample were “low” or “critically low” is concerning, and displays the need for higher quality research in TUD CPGs.

It would be logical to presume that SRs repeatedly being used as evidence underpinning recommendations across multiple guidelines would be of higher quality, but our data showed comparably low AMSTAR-2 assessment scores to that of the entire sample. Within our sample there are 64 (65.31%) SRs that were used as direct evidence for at least one specific TUD CPG recommendation. Of these 64 SRs, 14 were used as evidence to underpin specific recommendations in more than one TUD CPG (10 SRs in 2 separate CPGs; 4 SRs in 3 separate CPGs). Relative to the other SRs, these 14 are called upon as evidence more frequently, and thus carry more influence in the field of TUD management. Within these 14 frequently used SRs, 1 received a score of “moderate,” 11 received a “low” score, and 2 were scored as “critically low.” Lower than desired scores amongst the most frequently circulated evidence, while not completely at the fault of the authors, is concerning and shows the need for adherence to a validated standard of methodological quality appraisal. SRs are a summary of evidence available on a specific topic of interest, and the conduction and reporting of high-quality methodology is crucial to the validity of their attempted summary.²⁶ In the future, we recommend creators of TUD CPGs consider establishing a cut-off score for SRs using a validated quality appraisal tool. A cutoff standard for included SRs would ensure studies with inadequate methodological quality are not used to form clinical recommendations and would ultimately strengthen the justification of recommendations.

One possible method to raise quality assessment scores of SRs underpinning TUD CPG recommendations is to improve completeness of reporting. To assess reporting we conducted an appraisal of our sample using the PRISMA assessment. The final results of our PRISMA assessment were concerning. The 3 PRISMA items with the lowest percent completeness were items 1, 5, and 27. Item 1 asks if the title of the SR in question contains the term “systematic review,” “meta-analysis,” or both. It is important for researchers to include these phrases within the title to help subsequent researchers conduct efficient searches of databases. This specific item was complete in only 2% of Cochrane Reviews and almost 70% in all other SRs. While this standard of reporting is fundamental, Cochrane Reviews do not typically abide by this conventional measure given the implication that all of their studies are SRs, and they recommend screening their database when formulating a sample of studies. This specific item’s results may be skewed due to the Cochrane Groups deviation, but other reviewers and the Cochrane Group alike should consider including these phrases to benefit analytical research. Item 5, the least reported PRISMA item, asks if the study reports on and explains how to access their pre-constructed protocol and registration number. It is important to report these details to ensure there is no biased deviation from the study’s pre-planned methods, increasing the studies validity. The third least reported item, number 27, asks if the authors described the source of their funds and the roles of their funders. Reporting funders helps ensure findings are not economically driven and confirms the absence of significant conflicts of interest. These 3 under-reported PRISMA items could frequently be completed with only a few extra words within the article, but their absence brings potential bias and quality of these SRs into question. Addressing these reporting issues would help validate findings and improve the perceived quality of the studies at hand.

Clinical and Research Implications

Although more recent SRs were associated with higher PRISMA and AMSTAR-2 scores, our sample contained predominantly older SRs. Older SRs may skew clinical recommendations toward outdated modes of tobacco cessation while ignoring new developments that could be deemed harmful or beneficial. For example, there were only six CPGs examined that looked at nicotine replacement devices (NRD) as a mode of nicotine replacement therapy in attempts at smoking cessation.^27–32 All of these CPGs noted that NRD use as a mode of nicotine replacement therapy is inconclusive due to a lack of quality evidence as cited by four SRs. Two of these SRs were deemed “critically low” quality by the AMSTAR-2 assessment. The SRs used to underpin the CPGs NRD recommendation all used studies dating prior to the year 2000, which is concerning as most modern NRDs were not on the market until the mid 2000’s. In actuality, at least seven SRs have been conducted over the last year that address the use of NRDs for smoking cessation with conclusive results.^19,33–38 None of these unutilized, newer SRs used studies conducted prior to 2000. There is ample unused evidence that suggests NRDs for tobacco cessation is harmful and should not be recommended.^33–37 All the SRs used as reference for these harmful effects also had a statement advising clinicians not to make NRDs common clinical practice for smoking cessation due to the many emerging short-term side-effects and the lack of long-term studies conducted on this newer health hazard.^33–37 Thus, CPGs should cite this new evidence to make conclusive statements on the use or recommendation of NRDs in patients with TUD.

Another outdated area of concern within TUD CPGs is the lack of commentary on smartphones as a means for tobacco cessation. Smartphones and their applications have made advancements that have been deemed useful in many branches of medicine such as obesity, diabetes, and cardiac rehabilitation.^39–41 These advancements are another reason for more recent research to be included in CPGs. Rarely were smartphones/applications mentioned in TUD CPGs while other fields of study are now relying heavily on their use and influence.^39–41 A recent SR/MA conducted by Islam et al. showed smartphone use for weight loss had promising and emerging results.³⁹ Several higher quality SRs on smartphone interventions for tobacco cessation have been published in the past year that were not included in the current TUD CPGs.^20,42,43 Therefore, as TUD CPGs are updated the inclusion of emerging evidence is necessary.

Strengths and Limitations

To strengthen our results, extraction of SRs from CPGs was conducted in duplicate and all other extractions were conducted in triplicate. We also used PRISMA and AMSTAR-2 evaluation tools, which have been validated throughout medical literature.^23,44–47 However, a potential limitation to our study is the subjectivity of these assessments. An additional limitation is that PRISMA and AMSTAR-2 assessments were established after publication of some SRs in our sample. However, these older SRs are still used by TUD CPGs to inform recommendations and therefore must be assessed using the current standards.

Conclusion

Systematic Reviews used as evidence for TUD CPGs completed just 53.77% of the AMSTAR-2 quality assessment items and 74.68% of the PRISMA assessment on average. While the assessments themselves are imperfect, more thorough compliance increases the ability to interpret, reproduce, and validate the findings of a given study. Studies in our sample published by the Cochrane Group and those published more recently were associated with higher PRISMA and AMSTAR-2 assessment scores. Additionally, the lack of recently published SRs generates dated recommendations overlooking recent developments in TUD management. These findings are important given the clinical decisions, reimbursement patterns, and patient care outcomes that CPGs inform.⁴⁸ In their current state, recommendations in TUD CPGs are supported by SRs with suboptimal methodology and reporting, and should take action to improve their foundation of SRs used as evidence in future guideline formation.

Supplementary Material

A Contributorship Form detailing each author’s specific involvement with this content, as well as any supplementary data, are available online at https://academic.oup.com/ntr.

Declaration of Interests

Dr. Vassar reports receipt of funding from the National Institute on Drug Abuse, the National Institute on Alcohol Abuse and Alcoholism, the US Office of Research Integrity, Oklahoma Center for Advancement of Science and Technology, and internal grants from Oklahoma State University Center for Health Sciences. Dr. Hartwell reports funding from the National Institutes of Justice unrelated to the current topic. All other authors have no conflicts of interest to report — all outside of the present work.

Funding

This study was not funded.

Data Availability

All data in this study is available within the text, the associated tables, the online supplementary document, or upon reasonable request to the corresponding author.