Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jul 1.
Published in final edited form as: Psychiatr Serv. 2016 Feb 29;67(7):710–717. doi: 10.1176/appi.ps.201500234

Independent contractors in public mental health clinics: Implications for evidence-based practices and beyond

Rinad Beidas 1,, Rebecca Stewart 2, Courtney Benjamin Wolk 3, Danielle Adams 4, Steven C Marcus 5, Arthur Evans 6, Kamilah Jackson 7, Geoffrey Neimark 8, Matthew Hurford 9, Joan Erney 10, Ronnie Rubin 11, Trevor R Hadley 12, Frances Barg 13, David S Mandell 14
PMCID: PMC4930412  NIHMSID: NIHMS795580  PMID: 26927579

Abstract

Objectives

Community mental health clinics are increasingly utilizing independent contractors to provide clinical services. At the same time, many organizations are participating in initiatives intended to increase implementation of evidence-based practices (EBPs). The primary aim of this study was to understand the associations of utilizing independent contractors with clinician and organizational characteristics associated with implementation of EBPs; as well as to understand the potential impact of using independent contractors on mental health services delivery from the perspective of organizational leadership.

Methods

Quantitative data were collected from 130 therapists in 23 organizations; qualitative data were collected from executive administrators in 9 of the 16 organizations participating in EBP initiatives sponsored by the City of Philadelphia. Regression with random effects was used to estimate the associations between worker status (i.e., contractor, employee) and clinician attitudes towards EBPs, knowledge of EBPs, and organizational culture and climate. Qualitative inquiry was used to understand the impact of relying on independent contractors on organizational participation in EBP initiatives.

Results

Independent contractors endorsed less positive attitudes towards EBPs and scored lower on knowledge of EBPs. Interviews revealed four main themes: reasons for using independent contractors, general consequences of independent contractors, specific impact of independent contractors on participation in EBP initiatives, and suggestions for alternatives.

Conclusions

A growing number of community mental health clinics are relying on independent contractors. There may be consequences of this paradigm shift which deserve exploration.


Over the past fifteen years, there has been an increased emphasis on the crisis facing the behavioral health workforce, including lack of providers and concerns about effectiveness of services (1). An effort to address these issues was undertaken by the Annapolis Coalition on the Behavioral Health Workforce, culminating in a document outlining strategic goals (2) including broadening the concept of workforce, strengthening the workforce, and building structures to support the workforce (3). Concurrently and relatedly, the last fifteen years have also seen a growing interest in the implementation of evidence-based practices (EBPs) by community mental health clinics, largely in response to policy mandates (4). This is consistent with the recommendations made by the Annapolis Coalition to buttress the competencies of the behavioral workforce in EBPs (5).

One important matter related to the behavioral health workforce that has received little attention is the growing number of independent contractors in specialty community mental health (Manderscheid, personal communication, July 24th, 2015). Salaried employees receive benefits, and have productivity requirements, whereas independent contractors typically do not receive benefits or have productivity requirements. The organizational management literature suggests that utilizing independent contractors destabilizes the workforce because it may result in conflict between salaried employees and independent contractors (6) and increase turnover for both groups (7). In community mental health, the increasing reliance on independent contractors may have sequelae for both mental health services delivery and implementation of EBPs. Given that independent contractors are not employees per se, organizations may be less likely to invest in their professional development resulting in potentially poorer knowledge and attitudes towards EBPs. Relying on independent contractors may also affect organizational culture and climate by increasing stress within an organization. Conversely, the use of independent contractors may have a positive impact on services and/or EBP implementation because it may allow organizations to be more nimble in hiring individuals with the skillsets they deem necessary for the setting. Given that both clinician (i.e., knowledge and attitudes) and organizational characteristics (i.e., culture and climate) have been found to be predictors of implementation of EBPs (8, 9), it is important to explore the relationship between independent contractors and these variables.

The first aim of this study was to explore the associations between independent contractors and factors found to be associated with implementation of EBP (8, 9). No a priori hypotheses were specified. Using quantitative methods, we compared attitudes and knowledge of EBPs in independent contractor and salaried clinicians. Next, we examined organizational culture and climate while taking into account the ratio of independent contractors within each organization. The second aim of the study was to use qualitative methods to explore stakeholder perspectives on the impact of using independent contractors in a subset of organizations participating in EBP initiatives.

Method

Setting

The Philadelphia Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) has supported EBPs in the public mental health system through four initiatives: cognitive therapy (10), prolonged exposure, trauma-focused cognitive behavioral therapy, and dialectical behavior therapy. These initiatives include training and consultation in line with treatment developer recommendations and entail a significant allocation of resources on the part of clinicians, organizations, and the system.

Agencies

More than 100 community public mental health agencies in Philadelphia provide outpatient services to youth (Community Behavioral Health, personal communication, January 3rd, 2013). We used purposive sampling to recruit outpatient public mental health agencies which served the largest percentage of youth. Out of the total number of agencies, 29 served approximately 80% (n =23,354) of youth receiving publicly-funded mental health outpatient care, thus we targeted those agencies. Of these 29 agencies, 18 (62%) agreed to participate. Representatives of another organization participating in an EBP initiative asked to participate, resulting in a final sample of 19 agencies with 23 sites, 130 clinicians, and 36 supervisors. There were no exclusion criteria for staff. Approximately 60% (N=130) of therapists providing services in the 23 organizations participated in the study; 57% (n=74) of therapists were independent contractors.

Procedure

This research was approved by the City of Philadelphia and [masked] Institutional Review Boards. The leader of the organization was approached to solicit organizational participation. A two-hour meeting was scheduled, during which we obtained informed consent. All participants were compensated $50 and provided lunch. Quantitative data were collected in 2013.

The qualitative data were collected as part of a follow-up study where we conducted interviews with agency leadership to understand the process of participating in EBP initiatives sponsored by the City of Philadelphia (11). We used purposive sampling to specifically target agencies participating in EBP initiatives in our sample (K = 16) and completed 9 interviews. When approaching agencies, we asked them to identify leadership who would have an understanding of operations, workforce issues, and agency and system factors needed to support EBP initiatives. Data were collected in 2013–2014. Respondents received an additional $50 for participation.

Measures

Demographics were assessed using the Therapist Background Questionnaire (12). We also asked participants to report on worker status (i.e., independent contractor or salaried), hours worked weekly, and participation in DBHIDS EBP initiatives. Supervisors provided information on the number of clinicians in their unit and their worker status. From that information, we determined percentage of independent contractors. DBHIDS provided us with a list of organizations that had participated in EBP initiatives, the year they began participation, and that completion year (if applicable). From that information, we calculated the cumulative years each organization participated in DBHIDS EBP initiatives.

Dependent Variables

Clinician attitudes were assessed using the Evidence-Based Practice Attitude Scale (EBPAS; (13)), a self-report questionnaire that assesses appeal of EBP, requirements to use EBP, general openness to new practices, and divergence between EBP and usual practice. The EBPAS demonstrates good internal consistency and validity (14).

Clinician knowledge about EBP was measured using the Knowledge of Evidence-Based Services Questionnaire, a 40-item self-report instrument that measures general knowledge of EBP. Knowledge is measured on a continuum from 0 to 160, with higher scores indicative of more knowledge (KEBSQ; (15)). Psychometric data suggests temporal stability, discriminative validity, and sensitivity to training (15).

Organizational culture and climate were measured using the Organizational Social Context Measurement System (OSC; (16), a 105-item measure of the social context of mental health organizations. Organizational culture includes proficiency, rigidity, and resistance, and climate includes engagement, functionality, and stress. Proficient cultures are those in which clinicians prioritize the well-being of clients and are expected to be competent. Rigid cultures are those in which clinicians have little autonomy. Resistant cultures refer to ones in which clinicians are expected to be apathetic. Engaged climates refer to ones in which clinicians feel they can accomplish worthwhile things and remain invested. Functional climates are ones in which clinicians can get their job done effectively. Stressful climates refer to ones in which clinicians feel emotionally exhausted. Organizational culture and climate are measured with T-scores (mean of 50 and standard deviation of 10) based on a normed sample of 100 community mental health clinics (16). The OSC has strong psychometric properties (17).

Semi-structured interview

We developed a semi-structured interview guide to collect information about participants’ experiences with the four DBHIDS EBP initiatives focusing on inner (i.e., agency characteristics) and outer context (i.e., system characteristics) factors (11). We included follow-up questions specifically oriented towards understanding the relationships between utilizing independent contractors and the association with participation in EBP initiatives.

Data Analytic Plan

Quantitative analysis

Analyses were conducted using PROC MIXED and PROC REG in SAS 9.0. Missing data for predictor variables were less than 10%; series means were imputed for missing predictor variables. Dependent variable missing data were minimal (3%).

We used five mixed-effects linear regression models to evaluate the associations among clinician factors (i.e., worker status, age, hours worked per week, participation in EBP initiatives (yes/no)) and knowledge (i.e., total knowledge score as measured by the KEBSQ) and attitudes (EBPAS requirements appeal, openness, divergence). These mixed-effects models included random intercepts for organization and fixed effects for staff factors.

We also used six fixed-effects regression models in which the organization was the unit of observation to evaluate the associations among (a) the ratio of independent contractors and (b) cumulative years the organization participated in city-sponsored EBP initiatives, and organizational culture (proficiency, rigidity, and resistance) and climate (engagement, functionality and stress) as measured by the OSC. Organizational measures are included in analyses by aggregating individual responses within the organization (18, 19). On all organizational variables, both statistics were substantially above the suggested .60 level as measured by awg and/or rwg (19, 20); therefore participant responses to organizational constructs were averaged within each organization.

Qualitative

Transcripts were analyzed in an iterative process based upon modified grounded theory (2124). Through a close reading of eight transcripts, the investigators developed a set of codes that were applied to the data. A codebook was developed that included codes emerging from the transcripts (e.g., workforce) as well as a priori codes (e.g., barriers) derived from the original research questions which pertained to barriers and facilitators of participating in EBP initiatives. A subset of 12 transcripts (20%) was coded by two investigators, and inter-rater reliability was excellent (k = .98) (25). Through an inductive process, the first author independently read through the data to examine workforce themes related to independent contractors, producing memos including examples and commentary to reach consensus regarding newly-derived, emergent themes (22, 23). The second author read through and coded 2 (20%) of the same data for consensus.

Results

Quantitative Results

Table 1 provides demographic information about the clinicians who participated in the quantitative portion of the study. Table 2 presents the results of the analyses predicting attitudes and knowledge. Table 3 presents the results of the analyses predicting organizational culture and climate.

Table 1.

Clinician demographics (N = 130)a

Variable N %
Gender
  Male 30 23
  Female 99 76
  Transgender 1 1
Hispanic/Latinoa
  Yes 26 20
  No 97 75
Ethnicitya
  Asian 6 5
  Black or African American 27 22
  White 67 55
  Hispanic/Latino 13 11
  Multiracial 5 4
  Other 5 4
Academic backgrounda
  Bachelor’s degree 5 4
  Master’s degree 107 82
  Doctoral degree 12 9
Licensure statusa
  Yes 32 25
  No 51 39
  In process 42 32
Age (M±SD) 38.09±11.63
Years at current organization (M±SD) 3.35±4.65
a

Does not add up to 100% because of missing responses

Table 2.

Clinician models predicting knowledge and attitudes (N = 126)a

EBPAS
requirements
EBPAS
appeal
EBPAS
openness
EBPAS
divergence
KEBSQ
knowledge

Variable Predictor Estimates
Independent contractor
worker status (referent
= salaried)
.09 −.28* .02 −.16 −5.12*
Age −.01 −.01* −.01 .01 −.18*
Hours worked weekly −.02* −.01 .00 −.00 −.16*
Participated in EBP
initiative (referent
= did not participate)
.22 .02 .18 −.09 4.94*
a

EBPAS = Evidence Based Practices Attitude Scale - EBPAS subscales range from 0 (not at all) to 4 (very great extent); KEBSQ = Knowledge of Evidence Based Services Questionnaire - KEBSQ ranges from 0–160 where higher scores are indicative of more knowledge; EBP = evidence-based practice

*

= p <.05.

Table 3.

Organizational models predicting culture and climate [K = 23]a

Variable Proficiency Rigidity Resistance Engagement Functionality Stress
Ratio IC 7.54 −2.57 −4.75 1.87 9.40 −11.17^
Cumulative
years
participating
in EBP
initiative
1.77^ .56 .56 .19 1.85^ .04
a

IC = independent contractor; EBP = evidence-based practice

^

= p ≤.10; Each measure of organizational culture and climate is measured with T-scores with a mean of 50 and standard deviation of 10

Participation in EBP initiatives

We completed an ad hoc analysis using a logistic mixed effects model among organizations participating in EBP initiatives (K = 16), and found no association between worker status and participation in an EBP initiative. Of the therapists working in the 16 organizations participating in EBP initiatives, 41 participants participated; 43 did not.

Attitudes

Independent contractors had less favorable attitudes towards adopting EBPs. Specifically, they endorsed being less willing to adopt EBPs even if they found them appealing. Additionally, older clinicians were also less likely to adopt EBPs if they found them appealing. Clinicians who worked fewer hours per week had more open attitudes towards adopting EBPs if required.

Knowledge

We found that independent contractors, younger clinicians, clinicians who worked fewer hours per week, and clinicians not participating in EBP initiatives scored lower on knowledge of EBPs. Given that both worker status and participation in EBP initiatives were significant predictors of knowledge, we created an interaction term that combined the 2 variables (worker status*participation in EBP initiatives) to our model. The interaction term was not significant; therefore, we removed it.

Organizational culture and climate

Ratio of independent contractors and cumulative years participating in EBP initiative in the organization did not significantly predict proficiency, rigidity, resistance, engagement, functionality, or stress. A marginally significant association between cumulative years participating in EBP initiatives and proficiency (p=.06) and functionality (p=.10) was observed. A marginally significant negative association between proportion of independent contractor staff and stress was observed (p=.10).

Qualitative results

Participant characteristics

Six participants were female; three were male. Six participants were white, two participants were Hispanic/Latino, and one participant was American Indian/Alaskan Native. Mean age was 45.0±8.4. Participants came from both salaried organizations (defined as >50% of outpatient therapists were salaried; K = 4) and independent contractor organizations (defined as >50% of outpatient therapists were contractors; K = 5). Of the nine represented organizations, two had transitioned to using only independent contractors within the past year.

See Table 4 for relevant quotes. All individuals noted the financial difficulties of maintaining salaried employees in the current fiscal environment for mental health services, leading to the need for independent contractors. A minority of participants noted that independent contractors remove financial risk from the organization and places it on the therapist (i.e., they do not have to pay therapists if clients do not attend session). Participants also noted the general ramifications of independent contractors including turnover, employment instability, lack of connection to the organization, and potentially reduced quality of services.

Table 4.

Qualitative themes and illustrative examplesa

Theme N % Example
Reasons for switching to the independent contractor model
Financial difficulties
of the salaried model
6 67 Outpatient was so horribly underpaid so we
made that transition [to independent
contractors], and so now outpatient is not the
huge loss leader. I ran outpatient when it was
salaried and I cannot tell you the financial
stress of running that because your financial
goals were literally impossible, and you knew
no matter what would do, you were going to
be extremely under budget and looking at the
faces of folks that are working so very hard.
Contractor model
transfers financial risk
to the clinician
2 22 If a patient doesn’t come, the agency doesn’t
pay the clinician.
The organization wants to pass that fee on to a
contractor and say, ‘You know, we’ll support
you going, but we can’t pay for it.’ So, you’re
talking about travel expenses and all that, plus
the not being paid while they’re going to the
training.
Consequences of the independent contractor model (general)
Turnover 3 33 [The contractor model] does lend itself to that
kind of instability, kind of wondering. And
you know, and I have yet to have a contractor
give me proper notice.
Time = money 3 33 Contractor therapists are paid a piece rate,
which means that they see somebody for an
hour, and their rate is about 25.50 an hour.
And, if the person doesn’t show up, they get
nothing. Understood. They know that coming
in.
They work full time hours, and they are
making a lot of money, but in the back of their
mind they are thinking about the other stuff
they are doing that they are not getting paid
for (e.g., note writing, team meeting). I
recruited people who I can sense have a desire
to learn things because I figured that would
work best. But, it’s difficult, even if you have
a desire to learn things, you still have to eat.
Lack of
recognition/connection
to the organization
2 22 The [agency] sometimes forget about
contractors in a sense that they do a lot of nice
things for employees. Employees just got a
raise or they are getting ready to get a raise.
We have a day that we all get together and
celebrate and when we have that day,
contractors go home generally. You know, so
they are kind of left out, but at this point,
you’re looking at that being almost half the
agency.
People that are contractors are not connected
to the agency. So if you have a mission and
vision driven work, you’re going to have a
hard time making that connection, They don’t
go to all staff meetings, they don’t go to group
meetings; they come do the work, they go.
Uncertainty 1 11 You know, so that also limits the kind of
people that I can recruit because I have to find
someone who is able to maintain themselves
with no insurance and be able to consider that
they might not get all the money every week.
Reduced quality of services 2 22 You lose a quality element as well. I’m not
saying that independent contractors do poor
work, what I’m saying is that my ability to
develop and enhance and train individuals is
also very limited.
Consequences of the independent contractor model (EBP-specific)
Impacts staff selection 3 33 We only have contractors. So, the agency,
even if they do have money, because we do
have money for training, is not willing to
invest money to send a contractor to an
expensive training…that they may or may not
get a return on.
Contractors can not
meet initiative
requirements
4 44 But where the challenge has been is that we
have people who are contractors, and
therefore, their ability to commit time to the
project is limited.
Contractors have to be
committed to learning
EBP because they are
not paid for extra time
1 11 I have to find contracts who are willing to do
[EBP], which I am not paying them for,
including training and consultation, which
means ultimately they would just have to be
committed to the fact that they want to learn a
skill.
Incentive for
contractors to use EBP
1 11 Per diem staff [i.e., independent contractors]
are only paid if they’re doing a session. So,
they are going to do what works with the
people they see. There is an incentive there to
do that.
Alternatives or ways to improve the independent contractor model
Treating contractor
clinicians with
respect/providing
opportunities for
professional
development
5 56 We feel strongly enough about the initiative
and we want them to participate. We are
paying for their training time as per their
hourly fee-for-service rate. And they’re not
salaried employees. So the agency takes on a
certain amount of cost for that as well as some
reduced productivity because people spend
more time in supervision and beyond the
actual face-to-face session with the consumer.
What we have tried to do, the only thing we
changed [when we transitioned to the
contractor model] is how the staff are paid.
Other than that, we behave as a department
with staff who receives supervision.
Shared risk model 1 11 For employees we have tuition
reimbursement. When you sign up for tuition
reimbursement, you also sign off that you will
be at the agency for 2 years or you have to pay
the money back. So similarly, I think they
could include this in the contract of
independent contractor clinicians.
Primary care
integration
1 11 I think we’re going to keep the [salaried]
model because we believe in it and in addition
to that, it is going to be integrated in primary
care and behavioral health. We’re going to try
to go as far as we can with the health home
models.
a

EBP = evidence-based practices

Participants also noted the specific impact of independent contractors on participation in EBP initiatives. Respondents stated that they would not send contractors to EBP trainings because of the potential loss of investment due to contractors’ perceived transiency and turnover. Furthermore, the additional requirements of EBP initiatives (e.g., training and ongoing consultation) made it difficult to include contractors in these efforts.

A majority of participants shared their opinions of alternatives or improvements that could be made. Several participants noted that it is important to treat independent contractors with respect and provide them with professional development opportunities to increase their connection to the organization and reduce turnover. They gave examples including increasing the hourly rate of independent contractors trained in EBPs, paying for parking when independent contractors go to trainings, and providing clinical support. Some participants advocated paying a reduced training rate for independent contractors attending clinical support meetings.

Discussion

Nationally, a growing number of community mental health clinics are moving towards relying on independent contractors. We used mixed methods to begin to understand the associations between worker status and factors associated with implementation of EBPs (9). Our quantitative results suggest that independent contractors reported less positive attitudes towards EBPs and less knowledge than salaried clinicians. Our qualitative results provide preliminary insights into why some organizations are using independent contractors and consequences in the organizations we studied.

Our quantitative data suggest that independent contractors were not less likely than salaried therapists to participate in EBP initiatives; however this finding was not corroborated by the qualitative interviews. Furthermore, independent contractors demonstrated poorer knowledge of and attitudes towards EBPs, suggesting potentially less access to professional development opportunities. Administrators shared that their organizations would be less likely to include independent contractors in EBP initiatives because independent contractors were more likely to turnover (26) and less able to participate in initiative requirements (10, 27), suggesting that the increase in the number of organizations using independent contractors may pose a threat to implementation of EBPs. Although preliminary, these findings have important implications for the calls to action to better train the behavioral health workforce (13, 5, 28, 29). There may be a segment of the specialty provider workforce receiving less access to training opportunities. Raising awareness about this is an imperative for future efforts to support the behavioral health workforce. It is important to note that the clinician and organizational variables studies are only factors associated with implementation. Future research should investigate the impact of using independent contractors on actual implementation of EBPs.

The proportion of independent contractors in an organization did not affect organizational culture and climate. However, we were underpowered to detect small effects at the organizational level. For this reason, we included results significant at p < .1. These results suggest that organizations with more independent contractors may have less stressful climates. This finding may be explained by productivity requirements given that salaried staff are expected to bill a certain number of weekly patient hours whereas our informal conversations with stakeholders suggest that independent contractors decide how many hours they would like to work each week. Given the high no-show rate in community mental health (30), this may contribute to increased pressure to bill and/or carry a high case load which may be reflected in a stressful climate. Although the results do not indicate causation, it is important to attend to the role of climate as it is impacted by worker status given that organizational climate is predictive of job performance (31, 32) and client outcomes (33, 34).

A number of important preliminary insights were gleaned from the qualitative interviews. Participants discussed the rationale for using independent contractors and their perceptions of the impact of such a model on their organization and staff. Overwhelmingly, participants discussed the dire financial environment in community mental health (26, 35, 36), and the reality that they could not provide salary and benefits to their therapists. There may be alternatives to improve organizational fiscal health such as teaching organizations better business practices (37) or changing federal and state policies to support these organizations. The general consequences of relying on independent contractors were perceived as negative for both the organization and the staff. Stakeholders suggested that independent contractors were more likely to turnover, less likely to feel connected to their organization, and more likely to provide poorer clinical services.

A number of study limitations should be noted. First, and most importantly, we did not have information about therapist behavior, thus precluding us from drawing conclusions on the actual implementation of EBPs. Second, not all therapists in the organizations participated in the quantitative portion and we did not directly talk with therapists in the qualitative portion. Third, our interviews were conducted with a small number of organizations in Philadelphia, so the generalizability is unknown. Fourth, administrators may be biased towards their fellow employees (e.g., salary increases are tied to one another). Finally, we did not collect information regarding whether cases are assigned differentially to clinicians based on their employment status. Assigning cases according to the unique strengths of contractors and employees may resolve some of the concerns identified in this article.

Conclusions

The findings from this study have important implications. Very little has been written about the increasing number of independent contractors in the behavioral health workforce; in a report to Congress on mental health workforce issues, no mention was made of this phenomenon (35). Non-scholarly discourse around this topic has been plentiful, as many individuals, organizations, and systems working within and with specialty mental health outpatient services are impacted by this shift. It is our hope that this initial investigation launches a systematic research agenda that delineates the number and characteristics of independent contractors in the behavioral health workforce, the characteristics of organizations utilizing independent contractors, and the associations with organizational, clinician, client, and EBP implementation efforts. Although this is an important topic to examine from an implementation science lens, it has a much broader potential impact on mental health services delivery that is in critical need of study.

Acknowledgments

Disclosures and acknowledgements.

Disclosures. Dr. X receives royalties from Oxford University Press and has served as a consultant for Kinark Child and Family Services. Dr. ZZ has received grant support from Ortho-McNeil Janssen and Forest Research Institute and has served as a consultant to AstraZeneca and Alkermes.

Acknowledgements. We are especially grateful for the support that the Department of Behavioral Health and Intellectual disAbility Services has provided for this project, and for the Evidence Based Practice and Innovation (EPIC) group.

Funding. Funding for this research project was supported by the following grants from National Institute on Mental Health: (K23 MH099179, Dr. X; F32 MH103960, Dr. Y; F32 MH103955, Dr. Z). Additionally, the preparation of this article was supported in part by the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25 MH080916) and Quality Enhancement Research Initiative (QUERI), Department of Veterans Affairs Contract, Veterans Health Administration, Office of Research & Development, Health Services Research & Development Service. Dr. X was an IRI fellow from 2012–2014.

Contributor Information

Rinad Beidas, University of Pennsylvania - Psychiatry, 3535 Market Street 3015, Philadelphia, Pennsylvania 19104, rbeidas@upenn.edu.

Rebecca Stewart, University of Pennsylvania.

Courtney Benjamin Wolk, University of Pennsylvania - Psychiatry, Philadelphia, Pennsylvania.

Danielle Adams, University of Pennsylvania - Psychiatry, Center for Mental Health Policy and Services Research 3535 Market Street 3rd Floor, Philadelphia, Pennsylvania 19104.

Steven C. Marcus, University of Pennsylvania - School of Social Policy & Practice, Philadelphia, Pennsylvania

Arthur Evans, Philadelphia Department of Behavioral Health, Philadelphia, Pennsylvania.

Kamilah Jackson, Community Behavioral Health.

Geoffrey Neimark, Community Behavioral Health.

Matthew Hurford, Philadelphia Department of Behavioral Health and Intellectual disAbilities, Pennsylvania.

Joan Erney, Community Behavioral Health.

Ronnie Rubin, Community Behavioral Health, Philadelphia, Pennsylvania.

Trevor R. Hadley, Univ Of Pennsylvania - Ctr For Mental Health, 3600 Market St, Rm 717, Philadelphia, Pennsylvania 19104

Frances Barg, University of Pennsylvania.

David S. Mandell, University of Pennsylvania Perleman School of Medicine - Department of Psychiatry, Philadelphia, Pennsylvania

References

  • 1.Hoge MA, Stuart GW, Morris J, et al. Mental health and addiction workforce development: federal leadership is needed to address the growing crisis. Health Affairs. 2013;32:2005–2012. doi: 10.1377/hlthaff.2013.0541. [DOI] [PubMed] [Google Scholar]
  • 2.Hoge MA, Morris J, Daniels A, et al. An action plan on behavioral health workforce development. Rockville, Md: US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration; 2007. [Google Scholar]
  • 3.Hoge MA, Morris JA, Stuart GW, et al. A national action plan for workforce development in behavioral health. Psychiatric Services. 2009;60:883–887. doi: 10.1176/ps.2009.60.7.883. [DOI] [PubMed] [Google Scholar]
  • 4.Cooper JL, Aratani Y. The status of states' policies to support evidence-based practices in children's mental health. Psychiatric Services. 2009;60:1672–1675. doi: 10.1176/ps.2009.60.12.1672. [DOI] [PubMed] [Google Scholar]
  • 5.Hoge MA, Morris JA. Implementing best practices in behavioral health workforce education-building a change agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2004;32:85–89. doi: 10.1023/b:apih.0000042741.19527.a8. [DOI] [PubMed] [Google Scholar]
  • 6.Davis-Blake A, Uzzi B. Determinants of employment externalization - a study of temporary workers and independent contractors. Administrative Science Quarterly. 1993;38:195–223. [Google Scholar]
  • 7.Davis-Blake A, Broschak JP, George E. Happy together? How using nonstandard workers affects exit, voice, and loyalty among standard employees. Academy of Management Journal. 2003;46:475–485. [Google Scholar]
  • 8.Glisson C, Schoenwald SK, Hemmelgarn A, et al. Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology. 2010;78:537–550. doi: 10.1037/a0019160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Beidas RS, Marcus S, Aarons GA, et al. Predictors of community therapists' use of therapy techniques in a large public mental health system. JAMA Pediatrics. 2015;169:374–382. doi: 10.1001/jamapediatrics.2014.3736. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Stirman SW, Bhar SS, Spokas M, et al. Training and consultation in evidence-based psychosocial treatments in public mental health settings: the ACCESS Model. Professional Psychology-Research and Practice. 2010;41:48–56. doi: 10.1037/a0018099. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Beidas RS, Stewart RE, Adams DR, et al. A multi-level examination of stakeholder perspectives of implementation of evidence-based practices in a large urban publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-015-0705-2. in review. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Weisz J. Therapist Background Questionnaire. Los Angeles: University of California; 1997. [Google Scholar]
  • 13.Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: the Evidence-Based Practice Attitude Scale (EBPAS) Mental Health Services Research. 2004;6:61–74. doi: 10.1023/b:mhsr.0000024351.12294.65. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Aarons GA, Glisson C, Hoagwood K, et al. Psychometric properties and U.S. National norms of the Evidence-Based Practice Attitude Scale (EBPAS) Psychological Assessment. 2010;22:356–365. doi: 10.1037/a0019188. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Stumpf RE, Higa-McMillan CK, Chorpita BF. Implementation of evidence-based services for youth assessing provider knowledge. Behavior Modification. 2009;33:48–65. doi: 10.1177/0145445508322625. [DOI] [PubMed] [Google Scholar]
  • 16.Glisson C, Landsverk J, Schoenwald S, et al. Assessing the Organizational Social Context (OSC) of mental health services: implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35:98–113. doi: 10.1007/s10488-007-0148-5. [DOI] [PubMed] [Google Scholar]
  • 17.Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse & Neglect. 2012;36:621–632. doi: 10.1016/j.chiabu.2012.06.002. [DOI] [PubMed] [Google Scholar]
  • 18.James LR, Demaree RG, Wolf G. Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology. 1984;69:85–98. [Google Scholar]
  • 19.Brown RD, Hauenstein NMA. Interrater agreement reconsidered: an alternative to the r(wg) indices. Organizational Research Methods. 2005;8:165–184. [Google Scholar]
  • 20.Bliese P. Multilevel Theory, Research, and Methods in Organizations. San Francisco: Joseey-Bass; 2000. Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis. [Google Scholar]
  • 21.Beidas RS, Edmunds JM, Cannuscio CC, et al. Therapists perspectives on the effective elements of consultation following training. Administration and Policy in Mental Health and Mental Health Services Research. 2013;40:507–517. doi: 10.1007/s10488-013-0475-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Hill CE, Knox S, Thompson BJ, et al. Consensual qualitative research: an update. Journal of Counseling Psychology. 2005;52:196–205. [Google Scholar]
  • 23.Hill CE, Thompson BJ, Williams EN. A guide to conducting consensual qualitative research. Counseling Psychologist. 1997;25:517–572. [Google Scholar]
  • 24.Stirman SW, Miller CJ, Toder K, et al. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science. 2013;8:1–12. doi: 10.1186/1748-5908-8-65. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed] [Google Scholar]
  • 26.Beidas RS, Marcus S, Benjamin-Wolk C, et al. A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-015-0673-6. (Epub ahead of print, July 16, 2015) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Creed T, Stirman S, Evans A, et al. A model for implementation of cognitive therapy in community mental health: the Beck Initiative. The Behavior Therapist. 2014;37:56–65. [Google Scholar]
  • 28.Hewitt A, Larson S, Edelstein S, et al. A synthesis of direct service workforce demographics and challenges across intellectual/developmental disabilities, aging, physical disabilities, and behavioral health. Washington, DC: National Direct Service Workforce Resource Center; 2008. [Google Scholar]
  • 29.Hoge MA, Paris M, Jr, Adger H, Jr, et al. Workforce competencies in behavioral health: an overview. Administration and Policy in Mental Health and Mental Health Services Research. 2005;32:593–631. doi: 10.1007/s10488-005-3259-x. [DOI] [PubMed] [Google Scholar]
  • 30.Defife JA, Conklin CZ, Smith JM, et al. Psychotherapy appointment no-shows: rates and reasons. Psychotherapy. 2010;47:413–417. doi: 10.1037/a0021168. [DOI] [PubMed] [Google Scholar]
  • 31.Judge TA, Thoresen CJ, Bono JE, et al. The job satisfaction-job performance relationship: a qualitative and quantitative review. Psychological Bulletin. 2001;127:376–407. doi: 10.1037/0033-2909.127.3.376. [DOI] [PubMed] [Google Scholar]
  • 32.James LR, Choi CC, Ko CHE, et al. Organizational and psychological climate: a review of theory and research. European Journal of Work and Organizational Psychology. 2008;17:5–32. [Google Scholar]
  • 33.Glisson C, Green P. Organizational climate, services, and outcomes in child welfare systems. Child Abuse & Neglect. 2011;35:582–591. doi: 10.1016/j.chiabu.2011.04.009. [DOI] [PubMed] [Google Scholar]
  • 34.Williams NJ, Glisson C. Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: a United States national study. Child Abuse & Neglect. 2014;38:757–767. doi: 10.1016/j.chiabu.2013.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Hyde PS. Report to Congress on the Nation's Substance Abuse and Mental Health Workforce Issues. US Department for Health and Human Services, Substance Abuse and Mental Health Services Administration. 2013 [Google Scholar]
  • 36.Stewart R, Adams DR, Mandell D, et al. The perfect storm: collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services. doi: 10.1176/appi.ps.201500392. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Lloyd D. How to Maximize Service Capacity: Nuts and Bolts Solutions for Implementing Change in Behavioral Healthcare Organizations Below the Senior Management Level. Rockville, Md: National Council for Community Behavioral Healthcare; 1998. [Google Scholar]

RESOURCES