Table 2.

Themes and representative quotes

1. Perceptions of System Implementation Strategies
a. Important role of Training/technical assistance	i. Limited availability of training (Agency Leaders). “Training in a practice being available when it is needed. This has been a major issue with some practices. Certainly when you have the resignation of a fully trained staff, it has an impact.”
	ii. Training in multiple practices considered positive achievement (Agency Leaders). “Most of our current clinical staff are trained in an average of 3 Practices. We have done this over a period of 2 years. This gives us much greater flexibility with assigning cases and increasing access to care for clients.”
	iii. Need for ongoing funding for training (Agency Leaders). “(1) Financial support of Train-the-Trainer in PCIT. (2) Financial support for initial DBT training. (3) Financial support for MAP Train the Trainer (4) Authorization of existing [staff] to be authorized Train the Trainers of TFCBT, CBT, Seeking Safety.”
	iv. Anticipated reduction in training resources provided by LACDMH (System Leaders). “Regarding training funds, [LACDMH] explained that the funds to pay for training would decrease countywide, affecting all agencies…[LACDMH] encouraged the agency to network with other provider agencies to share costs of providing training to staff.”
b. Difficulty utilizing PEI Funding Allocation	i. Underutilization of PEI funds (Agency Leaders). “Based on PEI claims so far in FY 2012-13, it is projected that [the agency] will only use 44% of its PEI allocation…The number of clients and the amount of claims have both decreased. The agency explained that it hopes that its new access intake program and focus on getting PEI clients from outpatient clinics will result in increased utilization. Its capacity to implement PEI services has also been challenged by staff turnover and difficulty obtaining replacement training for PEI Practices.”
	ii. Outreach work needs more support as it is labor intensive and non-reimbursable (Agency Leaders). “Lastly outreach and engagement efforts lack the appropriate reimbursement rate by DMH to cover the necessary cost to implement the services to the fullest capacity and we request COS level rate as well as request advocacy from DMH to provide PEI services with providers to serve indigent populations.”
c. Challenges with outcome monitoring	i. Data entry time and resources a challenge (Agency Leaders). “Staff are finding that the dashboard updating and session planning aspects of MAP are time intensive and non-billable. Managing a large number of MAP cases can make carrying regular caseloads and billing percentages significantly less in this practice. This in turn lessens the number of families we are able to serve as a whole.”
	ii. Outcome measures not linguistically and culturally appropriate (Agency Leaders). “Since a good number of our clients do not read well, we need to read measures to them. We now have Outcomes Assistants who admin the measures (after training) and most of the time they need to read the questions/statements to the clients. The Spanish version of the YOQ is not very good.”
	iii. Outcome measures not necessarily capturing client progress (Agency Leaders). “Difficulty has arisen in obtaining the PEI (end of treatment) outcomes due to the youth being sporadically released without notice. As a result our data has not shown the true progress the youth have made.”
	iv. Consistent completion of outcome measures is challenging (System Leaders). “The timing of collecting outcome measures was discussed. Agency leadership reported trying to collect measures before the last session, finding that clients often do not attend the last appointment.”
d. Compliance with PEI Practice claiming allowances/PEI requirements	i. Timeline for treatment is limiting (Agency Leaders). Difficulty in maintaining fidelity to the models in regard to working within established time frames of treatment because of the uniqueness of the population we serve.
	ii. PEI targets are different population and different treatment length than what agencies used to (Agency Leaders). “Having the option of only selecting from the menu of DMH-approved PEI Practices, agency staff feel as though they are stuck with “fitting round pegs into square holes.” The agency acknowledged that many of its clients have already received therapy and want to learn how to move forward.”
	iii. Concerns about length of treatment (System Leaders). “The DMH Team advised the agency to closely monitor the fidelity with which it implements PEI Practices. For example, the average length of treatment for [EBP] was higher than the countywide average. In addition, at nearly 32 sessions, the average number of sessions per client was well over what is outlined in the model of 12-16 sessions.”
	iv. Concerns about ancillary services billed to PEI (System Leaders). “A number of claiming errors were noted. The agency was also advised to cease claiming to the No EBP and Unknown EBP codes.”
e. Guidelines developed during implementation	i. Confusion about PEI Implementation Requirements (Agency Leaders). “Communication about how and when to do things came in waves over time, without necessarily a central way of communicating. Sometimes there were some discrepancies between two parties (differences in outcomes measure implementation between CIMH & DMH OMA).”
2. Perceptions of PEI Practice Implementation
a. Practice Coverage	i. Generally, practices fit at least some of agency’s current populations appropriately. “Agency leadership selected PEI Practices that would fit best with its clientele. For example, it chose TF-CBT believing it would be good to address the trauma experienced by the marginalized population it serves. Although at first agency leadership was unsure if the Practice would work well because many parents have poor literacy skills, it finds the Practice a good fit for the agency.”
	ii. Need for more practices to fit populations not served. “The agency provided feedback that they need a PEI Practice that specifically addresses anxiety, and were looking into Individual CBT as an option. Unfortunately, Individual CBT is a PEI Practice that is only available to directly operated adult clinics.”
b. Pre-PEI supports for EBP implementation	i. Experience with EBPs facilitated PEI implementation. “With a long history and focus on treating trauma, [the agency] began implementing many Evidence Based Practices (EBPs) prior to the PEI transformation and viewed the PEI Transformation as an opportunity to grow. They are educated and involved with on-going developments of Practices through relationships with developers.”
c. Impact on staff	i. Change in therapist attitudes following initial implementation. “Although [agency] experienced some staff resistance to the transformation at the onset, staff has embraced the PEI program after observing improved outcomes in their clients and the agency reports an increased openness to the PEI Program among new staff.”
	ii. Adaptations and translations require more time and out of session work and place limits on staffing. “In addition, some materials for the PEI Practices are not available in the languages needed, which requires extra time and expense as staff members need to translate materials for clients.”
3. Types of Agency Implementation Strategies
a. Practice selection	i. Client needs and staff capacity both considered in selection of practices. “The agency takes clients’ cultural and linguistic needs into consideration when determining which and how many PEI Practices to train clinicians in at each site. Although clinicians believe it would be ideal to be trained in two to three (2-3) PEI Practices to avoid being overwhelmed, most clinicians are trained in four to five (4-5) practices so that they have enough tools/skills in order for the agency to be able to provide services to all age groups and meet client needs at each site.”
b. Integrating clinical/funding considerations in case assignments	i. Prioritizing fit between practice and client needs at intake. “Infrastructure was created for EBP assignment in which a triage team reviews each case and assigns an EBP based on symptoms. The clinicians then assess the clients and confirm the EBP. After the practice is implemented, clinicians present each case to a multidisciplinary team and describe how treatment is tailored to the client including the client’s culture 1 month after intake, after 6 months, and after a year to the ensure appropriate treatment is provided.”
c. Changing staffing	i. Creation of new positions “…the agency created positions for an EBP Coordinator, Agency-Wide Administrator (AWA) for each PEI Practice, and Site Coordinators that work cohesively to ensure fidelity of the PEI Practices models.”
	ii. Reallocating time in existing positions. “Recognizing the importance of PEI Program fidelity and the clinical utility of outcome measure data, the agency designated a Clinical Supervisor to monitor the PEI Program implementation and is in the process of identifying a full time employee to oversee data collection and entry.”
d. Infrastructure for implementation support	i. Increased structured opportunities for staff and management to communicate. “Leadership fosters a collaborative work environment that supports two-way communication between management and clinical staff. For example, management conducted a focus group to elicit feedback from senior clinicians regarding the translation and cultural adaptations required to successfully implement the PEI Program.”
	ii. Use of technology to facilitate PEI implementation. “The agency created a user-friendly Information System to monitor and track the implementation of its PEI Program internally. Reports are provided to staff monthly that include information by clinician on clients served, EBP session counts, core units and non-core units. The system generates automated reminders and administrators follow-up with clinicians when inconsistencies or core/non-core issues arise in the reports.”
e. Changes to clinical supervision procedures	i. More supervision time devoted to monitoring compliance with PEI requirements. “Outcomes are also discussed in supervision. In addition, supervisors are trained in all PEI Practices and review progress notes for fidelity of implementation.”
	ii. Addition of practice-specific supervision. “With a commitment to uphold PEI Program fidelity, Clinical Managers created a discussion forum by holding PEI Practice-specific consultation groups with an emphasis on sharing successful resources and techniques among the clinical team.”
f. Sustainment of practices	i. Use of Train the Trainer model. “When available, [the agency] uses a Train the Trainers model and focuses on staff mastery and model fidelity rather than on increasing the quantity of PEI Practices in which a clinician is trained.”
	ii. Implementing strategies to deter turnover. “[The agency] has low staff attrition, which has been a challenge for other agencies. The agency attributes this to their supportive, strengths-based, solution-focused approach. In addition, the agency closely screens applicants for experience and training in the PEI Practices offered by the agency in order to build a strong staff that fits well with its PEI Program.”
g. Increase outreach and engagement efforts	i. New linkages in community to drive referral sources. “The agency makes an effort to outreach to a new client population, with two (2) staff engaged in “grass roots” outreach efforts in non-traditional locations such as libraries, churches, Head Start programs, community fairs, and local clinics. In addition, the manager of the Birth to 5 program does outreach, and a local school asked the agency to make a presentation.”
	ii. Using multiple strategies to increase engagement “The agency provides bilingual and bicultural mental health services without the use of a translator. When parents and/or caregivers seek services, they can drop their children off at the “Recreation Neighborhood Center” fully equipped with a recreation room, dance studio, and weight room…In addition, the agency demonstrates its commitment to providing grassroots services by hiring and training local community residents.”
h. Adapt PEI practices based on age, developmental level, and culture	i. Adaptations made based on age, developmental level, and culture “Clinicians have found it necessary to adapt the delivery of some topics by involving the client in more psycho-education activities and/or role plays. For example, after the clinician uses chalk to write words on the sidewalk depicting healthy and unhealthy relationships, the client is asked to use chalk to circle only the words describing a healthy relationship.”