Skip to main content
Health Services Research logoLink to Health Services Research
. 2014 Dec 23;50(4):1125–1145. doi: 10.1111/1475-6773.12273

A Longitudinal Study of State Strategies and Policies to Accelerate Evidence-Based Practices in the Context of Systems Transformation

Traci Rieckmann 1,2,3,4,, Amanda Abraham 1,2,3,4, Janet Zwick 1,2,3,4, Caitlin Rasplica 1,2,3,4, Dennis McCarty 1,2,3,4
PMCID: PMC4545350  PMID: 25532616

Abstract

Objective

To profile state agency efforts to promote implementation of three evidence-based practices (EBPs): screening and brief intervention (SBIRT), psychosocial interventions, and medication-assisted treatment (MAT).

Data Sources/Study Setting

Primary data collected from representatives of 50 states and the District of Columbia’s Single State Authorities from 2007 to 2009.

Study Design/Data Collection

The study used mixed methods, in-depth, semistructured interviews and quantitative surveys. Interviews assessed state and provider strategies to accelerate implementation of EBPs.

Principal Findings

Statewide implementation of psychosocial interventions and MAT increased significantly over 3 years. In the first two assessments, states that contracted directly with providers were more likely to link use of EBPs to reimbursement, and states with indirect contract, through counties and other entities, increased recommendations, and some requirements for provision of specific EBPs. The number of states using legislation as a policy lever to promote EBPs was unchanged.

Conclusions

Health care reform and implementation of parity in coverage increases access to treatment for alcohol and drug use. Science-based substance abuse treatment will become even more crucial as payers seek consistent quality of care. This study provides baseline data on service delivery, contracting, and financing as state agencies and treatment providers prepare for implementation of the Affordable Care Act.

Keywords: Medication-assisted treatment, evidence-based practices, substance abuse


The economic and health-related costs of alcohol and drug use disorders (Buckley 2006; Rehm et al. 2009; Cai et al. 2010; Owens, Mutter, and Stocks 2010; Appleyard et al. 2011; Bouchery et al. 2011), coupled with incarceration and mortality rates (McNeil, Binder, and Robinson 2005; Buckley 2006; Hall et al. 2008; Hser et al. 2008; Kerr et al. 2008), burden individuals and communities. The 2009 National Survey on Drug Use and Health estimated that 22.5 million persons aged 12 or older met diagnostic criteria for substance dependence or abuse in the past year (Substance Abuse and Mental Health Services Administration [SAMHSA] 2010a). The estimated economic costs of illicit drugs are $194 billion (National Drug Intelligence Center 2011), and economic costs of excessive drinking are $223.5 billion (Bouchery et al. 2011). Thus, the public health and social and economic impacts associated with alcohol and drug use and abuse are substantial.

Specialty treatment centers (n ≈ 13,500), typically small, free-standing clinics provide the majority of the care for those struggling with substance use disorders (SUDs) in the United States (SAMHSA 2011a). Very few of these programs are affiliated with a hospital or have a primary care provider on staff, and the majority of these centers rely primarily on public sources of funding (SAMHSA 2011b). Therefore, the SUD service delivery system is typically isolated from primary care services and challenged to provide quality care with very limited funding and infrastructure.

The 2010 Patient Protection and Affordable Care Act (PPACA) along with the 2008 Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act (PL 110-343) are likely to increase coverage from public and commercial health plans, expanding services for patients with behavioral health disorders (“Mental Health Parity and Addiction Equity Act” 2008; “Patient Protection and Affordable Care Act of 2010” 2010). The Parity Act requires coverage of substance abuse treatment at parity or equivalent to medical/surgical benefits, increasing the opportunity for patients with behavioral health disorders to engage in and remain in care. Another significant advancement in terms of access to care, The Affordable Care Act (ACA), requires that treatments for SUDs be covered as essential benefits and supports the development of patient-centered medical homes that integrate primary care and treatment for SUDs disorders (Buck 2011). The National Drug Control Strategy, moreover, promotes increased integration with primary care and points toward the increased allocation of resources for community health centers (CHCs) or federally qualified health centers (FQHCs) to increase services for patients with SUDs (Office of National Drug Control Policy [ONDCP] 2011, 2012). Merging substance abuse treatment with general medical care requires new partnerships with medical organizations, including hospitals, physician’s offices, and CHCs (Carise 2010). As health care reform takes effect, the Substance Abuse and Mental Health Services Administration (Power 2010; SAMHSA 2010b) and others (Druss and Mauer 2010) advise that evidence-based practices (EBPs) must be disseminated and implemented across all aspects of the treatment system. Providers will be held accountable for providing quality services and improving outcomes. The expanded use of empirically supported services is one step in that direction.

The State’s Role in Addiction Treatment

Each state, and the District of Columbia, has a single state authority (SSA) charged with overseeing substance abuse treatment services. SSAs work with substate entities and directly or indirectly with publicly funded treatment providers to deliver statewide, regional, and local services. As purchasers of public sector treatment, SSAs can affect service delivery through policy mandates, budgetary support or reductions, infrastructure and administrative law changes, and by requiring or encouraging providers to use EBPs that integrate the best research evidence, clinical expertise, and patient values (Finnerty, Rapp, and Bond 2005; Rapp et al. 2005; Institute of Medicine 2006; Boyle 2009; Rieckmann et al. 2011; Knudsen and Abraham 2012). EBPs include both medication-assisted treatments (Ling et al. 2005; O’Malley et al. 2007; Krupitsky, Zvartau, and Woody 2010; Rösner et al. 2010, 2011) and behavioral therapies such as cognitive behavior therapy, couples and family therapy, and contingency management, that have been found to be effective interventions for many drug addictions (Rawson et al. 2006; Carroll and Onken 2007). Research has also shown that in states with requirements for comprehensive substance abuse assessment and related services, treatment programs are significantly more likely to offer wrap-around services such as family counseling and HIV/AIDS education (Chriqui et al. 2008). Thus, provider contracts or grants may include language about clinical care requirements, outcome reporting, and even the use of specific practices such as EBPs (Center for Substance Abuse Treatment [CSAT] 1998; Marton, Daigle, and de la Gueronniere 2005). Marton, Daigle, and de la Gueronniere (2005) identified contracts as an ideal medium for developing and defining SSA objectives to ensure quality care. Further, Humphreys and McLellan (2011) call for greater use of performance-based contracting and implementation of quality improvement strategies.

As the ACA is implemented, SSAs will be challenged to transition systems of care into more medically oriented settings to take advantage of the ACA and Medicaid coverage for the costs of care. The economic, political, and systems-wide feasibility of maintaining a separate specialty addiction treatment system seems remote, and small free-standing treatment facilities may struggle to remain economically viable (McLellan, Carise, and Kleber 2003). The most visible transition strategy is to link addiction treatment and behavioral health services closely with primary care and services offered in safety net clinics. Preparing for this shift, the Substance Abuse Prevention and Treatment Block Grant has evolved into a joint application for mental health and substance abuse (SAMHSA FY 2012-2013 Block Grant Application 2011).

The combined substance abuse and mental health block grant applications reflect the federal expectations for screening, services in primary care, and systems-wide change with substance abuse and behavioral health. Provider-level changes require state authorities to lead the change process as they are the primary overseers of service delivery. The Office of National Drug Control Policy (ONDCP) indicates that state government is the appropriate entity to partner with communities to expand prevention and integration efforts, review laws and regulations that impede recovery, and develop systems infrastructure and advance partnerships with other agencies and systems (i.e., criminal justice, health care, Medicaid/Medicare) (ONDCP 2011, 2012).

Quality Improvement and National Consensus Standards

The National Quality Forum’s (NQF) National Consensus Standards for Substance Use Disorders (NQF 2007) expands the 2005 interim report and asserts that treatment of SUDs involves a continuum of care and a long-term perspective based on a chronic care model (NQF 2007). Therapeutic interventions to treat substance use conditions, including psychosocial interventions and pharmacotherapy (i.e., medication-assisted treatment [MAT]), are identified as key practice domains (NQF 2007). The Standards also call for routine screening for SUDs in medical settings as screening, brief intervention, and referral to treatment (SBIRT) has gained substantial support over the past decade (Babor, McRee, and Kassebaum 2007). Recommended strategies for meeting these standards include financial incentives and mechanisms, use of regulations and accreditation, and infrastructure development (NQF 2005). Many are looking to SBIRT to address the treatment gap between those who are in need of treatment services and those who engage in services as it proves to be applicable to multiple settings (Bernstein et al. 2009; ONDCP 2012).

Improving access to evidence-based, efficacious services is an immediate challenge, yet research regarding effective state and provider strategies designed to improve access to the most effective and innovative practices in the context of the shifting health care environment is sparse. Interviews with SSA administrators completed in 2007, 2008, and 2009 provide a longitudinal perspective on strategies used to influence and promote implementation of the practices identified in the National Consensus Standards in publicly funded systems of care. We specifically note changes in legislation, contracting, and the SSA’s influence on MAT, psychosocial interventions, and SBIRT. Importantly, this study also serves as a baseline to document changes in service delivery, contracting, use of EBPs, and financing as state agencies and treatment providers prepare for full implementation of ACA.

Methods

This 3-year study utilized mixed methods, in-depth, semi-structured interviews and quantitative surveys with SSA representatives to capture the status of EBP implementation over time and the SSA strategies (e.g., use of contracting and state policies and other state-provider strategies) to increase adoption and implementation of empirically supported interventions. The Oregon Health & Science University Institutional Review Board reviewed and approved the study protocol.

Participants

SSA directors and designees from each of the 50 states and Washington, DC (N = 51) participated in the interviews. Recruitment sought participation from the individuals whose current position provided a unique understanding of the issues (Trochim and Donnelly 2006). The publically available Single State Agency Directory provided initial contact information and several Addiction Technology Transfer Centers (ATTCs) completed an initial rapid assessment interview in 2007 (see Rieckmann et al. 2009). Follow-up interviews were conducted by a team of trained interviewers in 2008 and 2009. Respondents for the 2007 interview (February to June 2007) included the SSA director or an assistant/deputy director (n = 31, 61 percent); or managers or treatment services director (n = 20, 39 percent). The 2008 and 2009 respondents were similar and included the SSA director or an assistant/deputy director (n = 36, 71 percent); or a manager or treatment services director (n = 15, 29 percent).

Instruments

Comprehensive interviews included brief quantitative surveys, descriptive items (i.e., infrastructure and specific polices), as well as open-ended qualitative questions. Items addressed organizational structure, authorization/licensure, treatment provider funding, regulations and legislation, staff functions related to EBPs, and implementation of EBPs. Each interview examined specific evidence-based domains and practices (see list in Table1) rated on a 1–5 Likert-type scale (1 = not at all implemented; 5 = extensively implemented). For example, psychosocial interventions items were introduced as follows: “please rate the extent that adoption of proven psychosocial interventions (e.g., Motivational Interviewing, Motivational Enhancement Therapy, Cognitive Behavioral Therapy, structured family and couples therapy, Contingency Management, Community Reinforcement Approach, 12-step facilitation therapy, and ASAM Patient Placement Criteria) has been implemented in your state.” For the psychosocial and MAT domains, respondents were first asked to rate the overall implementation of each domain and then asked to rate the implementation of specific EBPs within each domain. Interviews also asked respondents to describe their office’s structure, the SSA’s placement in state hierarchy, and whether the state authorities for mental health (SMHA) and substance abuse were colocated. Additional items probed for the presence of state legislation, state funding for EBPs, and the use of contract language to require EBPs. Documentation was requested regarding EBP-related legislation and contract language.

Table 1.

States Reporting Widespread Implementation§ of EBPs: 2007–2009

Evidence-Based Practices 2007 (N = 50) 2008 (N = 51) 2009 (N = 51)
M (SD) N (%) M (SD) N (%) M (SD) N (%)
Screening and brief intervention 2.48 (.97) 7 (13.7) 2.32 (.94) 6 (11.8) 2.41 (.92) 6 (11.8)
Overall psychosocial interventions 3.24 (.82) 16 (31.4) 3.84 (.82) 35 (68.6) 3.66 (.77) 32 (62.7)*,
Individual psychosocial interventions
 ASAM Placement Criteria 39 (76.5) 39 (76.5)
 MI/MET 30 (58.8) 40 (78.4)
 CBT 28 (54.9) 36 (70.6)
 Matrix model 8 (15.7) 11 (21.6)
 Seeking Safety 9 (17.6) 10 (19.6)
 DBT 7 (13.7) 9 (17.6)
 Overall medication-assisted treatment 2.62 (.99) 8 (15.7) 3.14 (1.01) 18 (35.3) 2.96 (.92) 15 (29.4)*,
Individual medication-assisted treatment
 Methadone 36 (70.6) 34 (66.7)
 Buprenorphine 11 (21.6) 7 (13.7)
 Oral naltrexone 3 (5.9) 6 (11.8)
 Disulfiram 5 (9.8) 6 (11.8)
 Injectable naltrexone 2 (3.9)
 Smoking cessation 18 (35.3)
*

Significant (p < .05) increase from 2007 to 2008.

Significant (p < .05) increase from 2008 to 2009.

Significant (p < .05) increase from 2007 to 2009.

§

Widespread implementation defined as 4 = considerably implemented or 5 = extensively implemented on the 5-point Likert-type scale vs. 1 = not at all implemented, 2 = slightly implemented, or 3 = moderately implemented.

Psychosocial interventions: ASAM = American Society of Addiction Medicine—Patient Placement Criteria; MI/MET = motivational interviewing/motivational enhancement therapy; CBT = cognitive behavioral therapy; DBT = dialectical behavior therapy.

Qualitative analysis of interview transcripts complemented the quantitative brief survey data. This concurrent mixed-methods approach allows a comprehensive profile of SSAs’ real-world experience with adoption and implementation of EBPs and assists with further expansion and clarification of findings that are not apparent when limited to one method of data collection (Bradley, Curry, and Devers 2007; Creswell and Plano Clark 2007; Creswell 2009). Health services research is increasingly using qualitative and mixed-methods research to examine diffusion of innovations (Greenhalgh et al. 2004; Crosson et al. 2005; Panzano et al. 2007; Rieckmann et al. 2009, 2011; Weiner et al. 2011).

Data Analysis

Descriptive statistics examined response distributions and relationships between variables of interest (SPSS version 18.0). Codes were added for regional location (U.S. Census Bureau regions) and the SSA’s location in the state hierarchy: Independent SSA (e.g., cabinet-level, reporting directly to Governor); under State Mental Health Authority (SMHA); or under umbrella agency (e.g., Department of Human Services). In addition to reporting means and standard deviations, the five-point scales used to assess implementation were dichotomized (1–3 vs. 4 or 5) to examine widespread implementation. Contracting strategies were coded into two categories: (1) direct contracts (SSA contracts directly with service providers or contracts through managed care entities) and (2) indirect contracts (contracts through substate entities, such as counties or local authorities). EBP language responses were dichotomized (contract language requires or encourages providers to use EBPs vs. no EBP contract language). For simplicity, the term “contract” included competitive grants.

Qualitative data were generated from digitally recorded interviews transcribed by an experienced transcriptionist and imported into Atlas.ti qualitative analysis software (Atlas.ti 2013). To ensure inter-rater reliability, we followed a four-step strategy in the analysis of these qualitative data. First, all coders were involved from the onset with the development and refinement of the coding scheme. Second, coders independently coded transcripts, and compared, reviewed, and discussed differences in coding decisions and rationale until achieving an understanding of codes and consistency in coding decisions. Third, investigators met with the coders to review consistency of coding. Fourth, 24 percent of documents were selected for “check-coding” and recoded by a separate analyst. Rater/check-rater consistency was assessed and 82 percent of the codes were consistent.

Results

Implementation of Evidence-Based Practices

Table1 describes the mean and standard deviation of implementation and the percentage of states that reported widespread implementation (4 or 5 on the five-point scale) of each evidence-based domain and specific practice within each domain.

Psychosocial Interventions

Over the 3-year study period, the percentage of states reporting widespread implementation of psychosocial interventions increased significantly (Table1). In addition, from 2007 to 2008, the overall mean for psychosocial interventions increased significantly from 3.24 (SD = .82) to 3.84 (SD = .82) and states reporting widespread implementation of psychosocial interventions more than doubled (t = 3.86; p < .01).

There are several reasons for the increase in widespread implementation of psychosocial EBPs from 2007 to 2008. Based on observations of transformations in the field, many states were beginning to respond to the national initiatives and federal and state funding efforts to address SUD treatment quality though increased use of EBPs and to tie quality and accountability to funding. As discussed above, in 2005 and 2007 the National Quality Form published their first two reports endorsing use of specific EBPs in substance abuse treatment to improve treatment quality (NQF 2005, 2007). In 2005, the NIDA Clinical Trials Network and SAMHSA launched their Blending Initiative to accelerate the dissemination of research-based drug abuse treatment findings into community-based practice, which included both national conferences and the dissemination of blending products focused on Motivational Interviewing in 2006 and Motivational Incentives in 2007 (Martino et al. 2010). Simultaneous advancements in technology and access to EBP materials have also increased in the last 10 years. Finally, state budget reductions required that states find new methods for allocating resources (rather than census and demonstrated need), which has led to a push toward increased accountability and documentation of impact. Qualitative interview data from states that reported increases in widespread implementation of psychosocial EBPs from 2007 to 2008 also support this finding. For example, three states began requiring treatment providers to use psychosocial EBPs to receive funding including ASAM PPC and CBT. Five states offered training to treatment providers in the use of psychosocial treatments (e.g., Matrix Model, CBT, Motivational Interviewing) and one state opened a school for Addiction Services that offers courses in psychosocial treatments.

Despite a slight decrease the following year, the significant increase held from 2007 to 2009 (t = 3.14; p < .01). When statewide implementation of individual psychosocial interventions was measured in 2008 and 2009, MI/MET was the only intervention to show a significant increase in widespread implementation (t = 2.42; p = .02). As of 2009, psychosocial interventions—in particular Cognitive Behavior Therapy, the American Society of Addiction Medicine Patient Placement Criteria, and Motivational Interviewing and Motivational Enhancement Therapy—were widely available in a majority of states.

Medication-Assisted Treatment

The mean and percentage of states reporting widespread overall implementation of MAT increased significantly over the 3-year study (Table1). From 2007 to 2008, states reporting widespread overall implementation of MAT more than doubled (t = 2.65; p = .01) and the significance increase held from 2007 to 2009 (t = 2.19; p = .03). When implementation of individual medications was measured in 2008 and 2009, no changes over time were observed. As of 2009, MAT was widely implemented in less than one-third of states, with SSA representatives reporting widespread implementation of only methadone in a majority of states.

Screening Brief Intervention and Referral to Treatment

There were no significant changes in implementation of SBIRT over the study period, with approximately 12 percent of SSAs reporting widespread implementation in each study year (Table1).

Funding Evidence-Based Practices

State funding of EBPs appeared to come from a variety of sources. Table2 describes funding for screening and brief intervention, psychosocial interventions, and MAT. Funding for SBIRT is balanced primarily across all four sources (general funds, Medicaid, Federal Block Grants, and other grants). Respondents indicated that psychosocial interventions were funded primarily by state general funds (96 percent) and federal block grants (94 percent). Interestingly, approximately three-fourths of states relied on state general funds and Medicaid to fund MAT.

Table 2.

Nationwide Funding of Evidence-Based Practices: 2009

EBP Category State General Funds Medicaid Federal SA Block Grants Other Grants
N (%) N (%) N (%) N (%)
Screening and brief intervention 22 (43.1) 21 (41.2) 16 (31.4) 22 (43.1)
Psychosocial interventions 49 (96.1) 40 (78.4) 48 (94.1) 26 (51.0)
Medication-assisted treatment 37 (72.5) 39 (76.5) 28 (54.9) 15 (29.4)

State Legislation

Use of legislation to promote EBPs was uncommon. As of 2007, five states reported a state legislative mandate regarding EBP implementation: Oregon, North Carolina, Alaska, Wisconsin, and Idaho (see Rieckmann et al. 2011). During the 3-year study, no new EBP-related legislative mandates became law (Table3).

Table 3.

Evidence-Based Practice Legislation and Provider Contracting: 2007 to 2009

2007 (N = 50) 2008 (N = 51) 2009 (N = 51)
N (%) N (%) N (%)
State has legislative mandate related to EBP implementation 5 (10.0) 5 (9.8) 5 (9.8)
Contracting method*
 Direct provider contracts 31 (62.0) 34 (66.7) 34 (66.7)
 Indirect funding to counties or other substate entities 19 (38.0) 17 (33.3) 17 (33.3)
Contract language*
 EBPs required or encouraged 30 (60.0) 32 (62.7) 33 (64.7)
 No EBP language 20 (40.0) 19 (37.3) 18 (35.3)
*

“Contracts” include competitive grants (RFAs, RFPs, RFRs) from the state.

Direct provider contracting includes several SSAs that used a managed care company to administer provider contracts under direction from the SSA.

Contracting

Contract language, conversely, was a common lever for EBP implementation (Table3). A majority of SSAs (62–67 percent) contracted directly with providers or through managed care organizations, vs. contracting indirectly through substate entities. States with direct contracting were significantly more likely to include EBP-related contract language in 2007 (77 percent; χ2 = 10.31, p < .01) and 2008 (74 percent; χ2 = 5.075, p = .05) as compared to states with indirect contracting (32 percent in 2007 and 41 percent in 2008). By 2009, the differences were no longer significant (68 percent vs. 59 percent) as EBP-related contract language increased significantly in states with indirect contracting. In addition, having contract language that required or encouraged EBPs was not associated with Census region, SSA colocation with SMHA, or SSA location in state hierarchy. Thus, it is likely that other factors not assessed in this study may influence the use of EBP language in contracts with providers.

Qualitative Themes

Thematic analysis suggests multiple factors limit implementation, and facilitating factors help the SSAs promote EBP implementation. Table4 describes the major themes and their impact on adoption of EBPs, from the perspective of SSA representatives. Workforce development (trainings and addressing attitudes and stigma), expanded funding, partnerships with medical settings and physicians (integration and collaboration), and enhanced infrastructure and leadership are facilitation themes which cross-cut each of the EBPs. Lack of funding for EBPs and workforce constraints (time commitment, lack of doctors, staff turnover, need for clinical supervision, need for training) were barriers for each EBP. In addition, workforce and community ideology was a barrier to implementation of MAT as was stigma among clients. Implementation of psychosocial EBPs was also limited by congruence with the client population and geography.

Table 4.

Adoption of Evidence-Based Practices: Barriers and Facilitating Factors

Barriers Facilitating Factors
Screening and brief intervention Lack of funding for implementation; inability to obtain grants to support SBIRT Collaborating with provider agencies to develop and strengthen grant ideas
Disinterest of the medical community; minimal support from Medicaid; stigma about patients with SUDs and hesitation of physicians Building relationships with primary care providers to increase education around SBIRT; holding conferences and trainings
Workforce constraints due to time commitment and intimidation of implementing SBIRT; need for training Restructure intra-agency organization to support implementation; coordinate internal strategic planning meetings to increase support
Psychosocial interventions Lack of funding for services and resources; loss of workforce due to financial constraints; perceived costs Utilize data systems to track progress on any particular psychosocial intervention
Incongruent practices for some client populations; geography Allowing the use of targeted case management services within substance abuse treatment
Time restrictions of the implementation process Contract with trainers and outreach for opportunities such as use of local ATTC offices and use of SAMHSA resources
Workforce and staffing needs; difficulty meeting the demands of continuous staff turnover Infrastructure changes that support training divisions as well as conferences within the department to highlight success and promote ongoing trainings
Need for clinical supervision and fidelity monitoring Increase clinical supervision and monitoring
Medication-assisted treatment Lack of financial resources and cost of medications; reimbursement complexities Large proportion of counseling workforce with Masters-level education
Lack of doctors willing to provide medications Maximized state-wide distribution of physicians prescribing buprenorphine
Workforce and community ideology and attitudes Absence of regulatory or legislative barriers that prevent use of MAT
Client-level discomfort about “swapping addictions” and potential shame from the public and patients about being a client of such a clinic Established workgroups and trainings for physicians and counselors about appropriateness of MAT

Discussion

Three years of data from SSAs for substance abuse in all 50 states and the District of Columbia reveal interesting relationships, critical changes, and areas of inertia. Our analyses suggest that states increased access to key EBPs, but there is room for improvement, especially with regard to screening and brief intervention and MATs other than methadone. While implementation of psychosocial interventions and MAT increased significantly, many states still lag in terms of consistent access to these services. It is noteworthy that many states managed to retain at least a moderate level of service access even during economic downturn. Because these data were collected during the initial phase of the recession (2007–2009), it is unclear if this trend has continued in the last 2 years as treatment resources have been reduced further in many states. Future research is needed to examine the impact of the recession on service access.

In terms of state strategies to drive change, our results are consistent with the literature and indicate that the majority of states have opted to promote EBPs by requiring or encouraging providers to deliver these practices with existing contract funds. Using contract language gives SSAs control over treatment delivery and is easier to change than other policy/regulatory options (CSAT 1998; Gelber and Rinaldo 2005; Marton, Daigle, and de la Gueronniere 2005; Rosenbloom et al. 2006; Finnerty et al. 2009). Indeed, very few states have opted to pass legislation to require EBPs (Rieckmann et al. 2011), in part because law is inflexible (Jacobson 2008) and state agencies do not control legislative decisions. However, this may change in the current environment of health reform, cost sharing, and payment reform.

This study emerged in part as a response to the NQF (2005, 2007) consensus standards and the critical and costly public health issues related to addiction. It is also timely as SSAs are responsible for the oversight and implementation of the federal block grants which fund the majority of specialty SUD treatment. As such they are charged with responding to ACA and a shifting environment with diminished resources and an unprepared workforce. Specifically, the SSAs’ role in the implementation of ACA may involve greater administration of Medicaid funding and services and collaboration with Medicaid agencies to organize and deliver treatment services as well as providing technical assistance and training for providers on Medicaid billing and reimbursement, integrating with mainstream health providers, and strategies for monitoring outcomes and performance.

Advancements in the field that continue to confirm that substance abuse is a chronic disease with similar integrative and ongoing care needs as diabetes and cardiovascular disease are also impacting service delivery. Comprehensive disease management (McLellan et al. 2005), the use of [multiple] MATs (Kaur, McQueen, and Jan 2008; Kranzler et al. 2010; Polsky et al. 2010; Baser et al. 2011), and psychosocial interventions (Glasner-Edwards and Rawson 2010) can reduce health care costs for those struggling with SUDs and state policy makers are increasingly pressed to respond to these findings.

SSAs also have a responsibility to ensure that they participate in the formation of Accountable Care Organizations in their regions and address the unique local challenges, government infrastructure, and stigma related to this chronic disease. SSAs have the capacity to partner with state and federal social service entities and primary care providers to create sharable data systems, licensed and competent providers, and a streamlined, efficient service delivery system. Some of these changes are supported by the ACA, which extends coverage through new provisions of the Medicaid program for current and newly eligible enrollees. The demand for improved outcomes, in part achieved through the implementation of EBPs, also correspond with both the Wellstone and Domenici Mental Health Parity and NQF (2005, 2007).

Strengths and Limitations

This longitudinal study obtained responses from all 50 states and the District of Columbia and therefore reflects the nation’s SSAs and their role in service delivery and policy implementation related to substance abuse treatment. Responses are subject to self-report bias and the accuracy of each respondent’s knowledge. However, participants were carefully chosen as SSA representatives who could speak knowledgably about state efforts to promote EBPs. Quantitative data included longitudinal information about implementation of psychosocial interventions and MAT, which is more complete than a cross-sectional study. Complementary data regarding contracting and policy efforts provide a more complete picture of state strategies to promote these practices. However, because surveys and interviews were refined during the 3-year study, longitudinal data are not available for all of our key variables of interest (e.g., individual EBPs, EBP funding), which limits examination of changes over time. However, using the variables available at all three time points (type of contract [direct/indirect], EBP language in contract [yes/no], and legislation policy in place or process regarding EBPs [yes/no], we estimated bivariate generalized estimating equations models which revealed a positive relationship between legislation in place/process and implementation of psychosocial EBPs (p < .05). Longitudinal analyses revealed no additional significant findings and are consistent with our prior work examining changes over time in implementation of MAT from 2007 and 2008 (Rieckmann et al. 2011). Qualitative data from in-depth interviews enrich the quantitative data and allow themes to emerge. Although states vary on many factors, they share common barriers and facilitating factors in their efforts to promote EBPs. Further research inclusive of patient-level data from each state would also make a significant contribution to the field and would complement state-level findings.

Implications

It is important to consider state agencies when examining publicly funded health services and the integration of behavior health with other service settings. Indeed, SSAs in each state have the potential to significantly impact services, the use of EBPs, and changes at the treatment provider level, which directly influence client outcomes. SSA representatives’ perceptions, beliefs, and preparation influence practice broadly and the use of EBPs specifically in their state. By promoting EBPs including psychosocial interventions and MAT, states can continue to improve quality of care for substance abuse treatment clients and address the triple aim (i.e., improving the patient care experience, improving population health, and reducing the per capita cost of health care) as well as equity in delivery of services (Berwick, Nolan, and Whittington 2008).

Acknowledgments

Joint Acknowledgment/Disclosure Statement: An award from the Robert Wood Johnson Foundation (award number) supported the collection and analysis of study data. Traci Rieckmann receives support through awards from the Substance Abuse and Mental Health Services Administration (ATTC award number) and the National Institute on Drug Abuse (R33 DA035640). Dennis McCarty receives support through awards from the National Institute on Drug Abuse (U10 DA015815, P50 DA018165, R01 DA030431, and R01 DA029716); he also serves as the Principal Investigator on research service agreements with Alkermes, Inc., and Purdue Pharma, LLC.

Disclosures: None.

Disclaimers: None.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

hesr0050-1125-sd1.pdf (1.2MB, pdf)

References

  1. Appleyard K, Berlin LJ, Rosanbalm KD. Dodge KA. Preventing Early Child Maltreatment: Implications from a Longitudinal Study of Maternal Abuse History, Substance Use Problems, and Offspring Victimization. Prevention Science. 2011;12:139–49. doi: 10.1007/s11121-010-0193-2. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Atlas.ti. Version 6.2. [Computer Software] Berlin: Scientific Software Development; 2013. [Google Scholar]
  3. Babor TF, McRee BG. Kassebaum PA. Screening, Brief Intervention, and Referral to Treatment (SBIRT): Toward a Public Health Approach to the Management of Substance Abuse. Substance Abuse. 2007;28:7–30. doi: 10.1300/J465v28n03_03. [DOI] [PubMed] [Google Scholar]
  4. Baser O, Chalk M, Fiellin DA. Gastfriend DR. Cost and Utilization Outcomes of Opioid-Dependence Treatments. American Journal of Managed Care. 2011;17(8):S235–48. [PubMed] [Google Scholar]
  5. Bernstein E, Bernstein JA, Stein JB. Saitz R. SBIRT in Emergency Care Settings: Are We Ready to Take It to Scale? Academic Emergency Medicine. 2009;16:1072–7. doi: 10.1111/j.1553-2712.2009.00549.x. “ ”. [DOI] [PubMed] [Google Scholar]
  6. Berwick DM, Nolan TW. Whittington J. The Triple Aim: Care, Health, and Cost. Health Affairs. 2008;27(3):759–69. doi: 10.1377/hlthaff.27.3.759. [DOI] [PubMed] [Google Scholar]
  7. Bouchery EE, Harwood HJ, Sacks JJ, Simon CJ. Brewer RD. Economic Costs of Excessive Alcohol Consumption in the U.S., 2006. American Journal of Preventive Medicine. 2011;41(5):516–24. doi: 10.1016/j.amepre.2011.06.045. [DOI] [PubMed] [Google Scholar]
  8. Boyle M. Response 2: The States Are the Key Factor in Successful EBP Adoption and Technology Transfer. The Bridge. 2009;1(2) [accessed on Dec. 2, 2012]. Available at http://www.attcnetwork.org/find/news/attcnews/epubs/v1i2_article02.asp. [Google Scholar]
  9. Bradley EH, Curry LA. Devers KJ. Qualitative Data Analysis for Health Services Research: Developing Taxonomy, Themes, and Theory. Health Services Research. 2007;42:1758–72. doi: 10.1111/j.1475-6773.2006.00684.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Buck JA. The Looming Expansion and Transformation of Public Substance Abuse Treatment under the Affordable Care Act. Health Affairs. 2011;30(8):1402–10. doi: 10.1377/hlthaff.2011.0480. [DOI] [PubMed] [Google Scholar]
  11. Buckley PF. Prevalence and Consequences of the Dual Diagnosis of Substance Abuse and Severe Mental Illness. Journal of Clinical Psychiatry. 2006;67(Suppl 7):5–9. [PubMed] [Google Scholar]
  12. Cai R, Crane E, Poneleit K. Paulozzi L. Emergency Department Visits Involving Nonmedical Use of Selected Prescription Drugs in the United States, 2004-2008. Journal of Pain and Palliative Care Pharmacotherapy. 2010;24:293–7. doi: 10.3109/15360288.2010.503730. [DOI] [PubMed] [Google Scholar]
  13. Carise D. 2010. “ What Healthcare Reform Means for Substance Abuse Treatment.” [accessed on February 18, 2011]. Available at http://www.huffingtonpost.com/deni-carise/what-healthcare-reform-me_b_514029.html.
  14. Carroll KM. Onken LS. Behavioral Therapies for Drug Abuse. The Journal of Lifelong Learning in Psychiatry. 2007;5(2):240–8. doi: 10.1176/appi.ajp.162.8.1452. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Center for Substance Abuse Treatment [CSAT] Contracting for Managed Substance Abuse and Mental Health Services: A Guide for Public Purchasers (TAP 22) Rockville, MD: Substance Abuse and Mental Health Services Administration; 1998. [Google Scholar]
  16. Chriqui JF, Terry-McElrath Y, McBride DC. Eidson SkS. State Policies Matter: The Case of Outpatient Drug Treatment Program Practices. Journal of Substance Abuse Treatment. 2008;35:13–21. doi: 10.1016/j.jsat.2007.08.012. [DOI] [PubMed] [Google Scholar]
  17. Creswell JW. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage; 2009. [Google Scholar]
  18. Creswell JW. Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage; 2007. [Google Scholar]
  19. Crosson JC, Stroebel C, Scott JG, Stello B. Crabtree BF. Implementing an Electronic Medical Record in a Family Medicine Practice: Communication, Decision Making, and Conflict. Annals of Family Medicine. 2005;3(4):307–11. doi: 10.1370/afm.326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Druss BG. Mauer BJ. Health Care Reform and Care at the Behavioral Health-Primary Care Interface. Psychiatric Services. 2010;61:1087–92. doi: 10.1176/ps.2010.61.11.1087. [DOI] [PubMed] [Google Scholar]
  21. Finnerty M, Rapp C. Bond G. State Health Authority Yardstick (SHAY): Impact of State Level Action on the Quality and Penetration of EBPs in the Community. Baltimore, MD: NRI Conference on State Mental Health Agency Services Research, Program Evaluation, and Policy; 2005. [Google Scholar]
  22. Finnerty MT, Rapp CA, Bond GR, Lynde DW, Ganju V. Goldman HH. The State Health Authority Yardstick. Community Mental Health Journal. 2009;45:228–36. doi: 10.1007/s10597-009-9181-z. [DOI] [PubMed] [Google Scholar]
  23. Gelber S. Rinaldo D. State Substance Abuse Agencies and Their Placement within Government: Impact on Organizational Performance and Collaboration in 12 States. Berkeley, CA: The Avisa Group; 2005. [Google Scholar]
  24. Glasner-Edwards S. Rawson R. Evidence-Based Practices in Addiction Treatment: Review and Recommendations for Public Policy. Health Policy. 2010;97(2–3):93–104. doi: 10.1016/j.healthpol.2010.05.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Greenhalgh T, Robert G, MacFarlane F, Bate P. Kyriakido O. Diffusion of Innovations in Services Organizations: Systematic Review and Recommendations. Milbank Quarterly. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Hall AJ, Logan JE, Toblin RL, Kaplan JA, Kraner JC, Bixler D, Crosby AE. Paulozzi LJ. Patterns of Abuse among Unintentional Pharmaceutical Overdose Fatalities. Journal of the American Medical Association. 2008;300(22):2613–20. doi: 10.1001/jama.2008.802. [DOI] [PubMed] [Google Scholar]
  27. H.R. 3590–111th Congress. 2010. “ The Patient Protection and Affordable Care Act ” [accessed on October 28, 2013]. Available at http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/pdf/PLAW-111publ148.pdf.
  28. H.R. 6983–110th Congress. 2008. “ Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act of 2008 ” [accessed on October 28, 2013]. Available at http://www.govtrack.us/congress/bills/110/hr6983.
  29. Hser YI, Huang D, Brecht ML, Li L. Evans E. Contrasting Trajectories of Heroin, Cocaine, and Methamphetamine Use. Journal of Addictive Disease. 2008;27(3):13–21. doi: 10.1080/10550880802122554. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Humphreys K. McLellan AT. A Policy-Oriented Review of Strategies for Improving the Outcomes of Services for Substance Use Disorder Patients. Addiction. 2011;106(12):2058–66. doi: 10.1111/j.1360-0443.2011.03464.x. [DOI] [PubMed] [Google Scholar]
  31. Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC: National Academies Press; 2006. [PubMed] [Google Scholar]
  32. Jacobson PD. Transforming Clinical Practice Guidelines into Legislative Mandates: Proceed with Abundant Caution (Commentary) Journal of the American Medical Association. 2008;299:208–10. doi: 10.1001/jama.2007.12. [DOI] [PubMed] [Google Scholar]
  33. Kaur AD, McQueen A. Jan S. Opioid Drug Utilization and Cost Outcomes Associated with the Use of Buprenorphine-Naloxone in Patients with a History of Prescription Opioid Use. Journal of Managed Care Pharmacy. 2008;14(2):186–94. doi: 10.18553/jmcp.2008.14.2.186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Kerr T, Small W, Johnston C, Li K, Montaner JS. Wood E. Characteristics of Injection Drug Users Who Participate in Drug Dealing: Implications for Drug Policy. Journal of Psychoactive Drugs. 2008;40(2):147–52. doi: 10.1080/02791072.2008.10400624. [DOI] [PubMed] [Google Scholar]
  35. Knudsen HK. Abraham AJ. Perceptions of the State Policy Environment and Adoption of Medications in the Treatment of Substance Use Disorders. Psychiatric Services. 2012;63:19–25. doi: 10.1176/appi.ps.201100034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Kranzler HR, Montejano LB, Stephenson JJ, Wang S. Gastfriend DR. Effects of Naltrexone Treatment for Alcohol-Related Disorders on Healthcare Costs in an Insured Population. Alcoholism: Clinical and Experimental Research. 2010;34(6):1090–7. doi: 10.1111/j.1530-0277.2010.01185.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Krupitsky E, Zvartau E. Woody G. Use of Naltrexone to Treat Opioid Addiction in a Country in Which Methadone and Buprenorphine Are Not Available. Current Psychiatry Reports. 2010;12:448–53. doi: 10.1007/s11920-010-0135-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Ling W, Amass L, Shoptaw S, Annon JJ, Hillhouse M, Babcock D, Brigham G, Harrer J, Reid M, Muir J, Buchan B, Orr D, Woody G, Krejci J, Ziedonis D the Buprenorphine Study Protocol Group. A Multi-Center Randomized Trial of Buprenorphine-Naloxone versus Clonidine for Opioid Detoxification: Findings from the National Institute on Drug Abuse Clinical Trials Network. Addiction. 2005;100:1090–100. doi: 10.1111/j.1360-0443.2005.01154.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Martino S, Brigham GS, Higgins C, Gallon S, Freese TE, Albright LM, Hulsey EG, Krom L, Storti SA, Perl H, Nugent CD, Pintello D. Condon TP. Partnerships and Pathways of Dissemination: The National Institute on Drug Abuse—Substance Abuse and Mental Health Services Administration Blending Initiative in the Clinical Trials Network. Journal of Substance Abuse Treatment. 2010;38:S31–43. doi: 10.1016/j.jsat.2009.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Marton A, Daigle J. de la Gueronniere G. Identifying State Purchasing Levers for Promoting the Use of Evidence-Based Practice in Substance Abuse Treatment. Hamilton, NJ: Center for Health Care Strategies; 2005. [Google Scholar]
  41. McLellan AT, Carise D. Kleber HD. Can the National Addiction Treatment Infrastructure Support the Public’s Demand for Quality Care? Journal of Substance Abuse Treatment. 2003;25:117–21. “ ”. [PubMed] [Google Scholar]
  42. McLellan AT, Weinstein RL, Shen Q, Kendig C. Levine M. Improving Continuity of Care in a Public Addiction Treatment System with Clinical Case Management. American Journal on Addictions. 2005;14(5):1–15. doi: 10.1080/10550490500247099. [DOI] [PubMed] [Google Scholar]
  43. McNeil DE, Binder RL. Robinson JC. Incarceration Associated with Homelessness, Mental Disorder, and Co-Occurring Substance Abuse. Psychiatric Services. 2005;56:840–6. doi: 10.1176/appi.ps.56.7.840. [DOI] [PubMed] [Google Scholar]
  44. National Drug Intelligence Center. The Economic Impact of Illicit Drug Use on American Society. Washington, DC: U.S. Department of Justice; 2011. [Google Scholar]
  45. National Quality Forum [NQF] Evidence-Based Treatment Practices for Substance Use Disorders, Workshop Proceedings. Washington, DC: National Quality Forum; 2005. [Google Scholar]
  46. National Quality Forum [NQF] National Voluntary Consensus Standards for the Treatment of Substance Use Conditions: Evidence-Based Treatment Practices. Washington, DC: National Quality Forum; 2007. [Google Scholar]
  47. Office of National Drug Control Policy. The National Drug Control Strategy, 2011. Washington, DC: The White House; 2011. [Google Scholar]
  48. Office of National Drug Control Policy. The National Drug Control Strategy, 2012. Washington, DC: The White House; 2012. [Google Scholar]
  49. O’Malley SS, Garbutt JC, Gastfriend DR, Dong Q. Kranzler HR. Efficacy of Extended-Release Naltrexone in Alcohol-Dependent Patients Who Are Abstinent before Treatment. Journal of Clinical Psycho-Pharmacology. 2007;27:507–12. doi: 10.1097/jcp.0b013e31814ce50d. [DOI] [PubMed] [Google Scholar]
  50. Owens PL, Mutter R. Stocks C. Mental Health and Substance Abuse-Related Emergency Department Visits among Adults. Agency for Healthcare Research and Quality; 2010. Statistical Brief 92. [PubMed] [Google Scholar]
  51. Panzano PC, Roth D, Sweeney HA, Massatti R, Carstens C, Seffrin B. Bunt E. The Innovation Diffusion and Adoption Research Project (IDARP): A Process Overview and a Preview of Qualitative Data from Interviews. In: Roth D, Lutz WJ, editors; New Research in Mental Health. Columbus, OH: Ohio Department of Mental Health; 2007. pp. 78–89. Vol 17, [Google Scholar]
  52. Polsky D, Glick HA, Yang J, Subramaniam GA, Poole SA. Woody GE. Cost-Effectiveness of Extended Buprenorphine-Naloxone Treatment for Opioid-Dependent Youth: Data from a Randomized Trial. Addiction. 2010;105(9):1616–24. doi: 10.1111/j.1360-0443.2010.03001.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Power AK. Transforming the Nation’s Health: Next Steps in Mental Health Promotion. American Journal of Public Health. 2010;100:2343–6. doi: 10.2105/AJPH.2010.192138. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Rapp CA, Bond GR, Becker DR, Carpinello SE, Nikkel RE. Gintoli G. The Role of State Mental Health Authorities in Promoting Improved Client Outcomes through Evidence-Based Practice. Community Mental Health Journal. 2005;41:347–62. doi: 10.1007/s10597-005-5008-8. [DOI] [PubMed] [Google Scholar]
  55. Rawson RA, McCann MJ, Flammino F, Shoptaw S, Miotto K, Reiber C. Ling W. A Comparison of Contingency Management and Cognitive-Behavioral Approaches for Stimulant-Dependent Individuals. Addiction. 2006;101:267–74. doi: 10.1111/j.1360-0443.2006.01312.x. [DOI] [PubMed] [Google Scholar]
  56. Rehm J, Mathers C, Popova S, Thavorncharoensap M, Teerawattananon Y. Patra J. Global Burden of Disease and Injury and Economic Cost Attributable to Alcohol Use and Alcohol-Use Disorders. Lancet. 2009;373(9682):2223–33. doi: 10.1016/S0140-6736(09)60746-7. [DOI] [PubMed] [Google Scholar]
  57. Rieckmann TR, Kovas AE, Fussell HE. Stettler NM. Implementation of Evidence-Based Practices for Treatment of Alcohol and Drug Disorders: The Role of the State Authority. Journal of Behavioral Health Services and Research. 2009;36(4):407–19. doi: 10.1007/s11414-008-9122-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Rieckmann TR, Kovas AE, Cassidy EF. McCarty D. Employing Policy and Purchasing Levers to Increase the Use of Evidence-Based Practices in Community-Based Substance Abuse Treatment Settings: Reports from Single State Authorities. Evaluation and Program Planning. 2011;34(4):366–74. doi: 10.1016/j.evalprogplan.2011.02.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Rosenbloom DL, Leis RG, Shah P. Ambrogi R. Blueprint for the States: Policies to Improve the Way States Organize and Deliver Alcohol and Drug Prevention and Treatment: The Findings of a National Policy Panel. Boston, MA: Join Together; 2006. [Google Scholar]
  60. Rösner S, Hackl-Herrwerth A, Leucht S, Vecchi S, Srisurapanont M. Soyka M. Opioid Antagonists for Alcohol Dependence. Cochrane Database of Systematic Reviews. 2010;12:CD001867. doi: 10.1002/14651858.CD001867.pub3. doi: 10.1002/14651858.CD001867.pub3. [DOI] [PubMed] [Google Scholar]
  61. Rösner S, Hackl-Herrwerth A, Leucht S, Lehert P, Vecchi S. Soyka M. Acamprosate for Alcohol Dependence. Cochrane Database of Systematic Reviews. 2011;9:CD004332. doi: 10.1002/14651858.CD004332.pub2. doi: 10.1002/14651858.CD004332.pub2. [DOI] [PubMed] [Google Scholar]
  62. Substance Abuse and Mental Health Services Administration. 2010a. “ Description of a Good and Modern Addictions and Mental Health Service System (Draft – December 20, 2010) ” [accessed on February 18, 2011]. Available at http://www.dshs.wa.gov/pdf/dbhr/good_and_modern_12_20_2010_508.pdf.
  63. Substance Abuse and Mental Health Services Administration. Results from the 2009 National Survey on Drug Use and Health: National Findings (Office of Applied Studies, NSDUH Series H-38A, HHS Publication No. SMA 10-4856 Findings) Rockville, MD: Department of Health & Human Services, Substance Abuse and Mental Health Services Administration; 2010b. [Google Scholar]
  64. Substance Abuse and Mental Health Services Administration. 2011a. “ FY 2012-2013 Block Grant Application: Community Mental Health Plan and Report, Substance Abuse Prevention and Treatment Plan and Report ” [accessed on July 13, 2011]. Available at http://www.samhsa.gov/sites/default/files/bg_planning_section.pdf.
  65. Substance Abuse and Mental Health Services Administration. 2011b. “ National Registry of Evidence-Based Programs and Practices ” [accessed on February 22, 2011]. Available at http://nrepp.samhsa.gov.
  66. Trochim W. Donnelly JP. The Research Methods Knowledge Base. 3rd Edition. Mason, OH: Atomicdog Publishing; 2006. [Google Scholar]
  67. Weiner BJ, Amick HR, Lund JL, Lee SD. Hoff TJ. Use of Qualitative Methods in Published Health Services and Management Research: A 10-Year Review. Medical Care Research and Review. 2011;68:3–33. doi: 10.1177/1077558710372810. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Appendix SA1: Author Matrix.

hesr0050-1125-sd1.pdf (1.2MB, pdf)

Articles from Health Services Research are provided here courtesy of Health Research & Educational Trust

RESOURCES