Translating Cognitive Behavioral Interventions from Bench to Bedside: The feasibility and acceptability of cognitive remediation in research as compared to clinical settings

Abstract

Cognitive Remediation (CR) research typically addresses internal validity, and few studies consider CR in a real-world context. This study evaluated the fit between the program conditions and treatment model in research and clinical settings, with the goal of informing future research on the contextual challenges associated with the implementation of CR. Data was drawn from an initiative by New York State’s Office of Mental Health (OMH), to implement CR programs for adults with Serious Mental Illness (SMI) in 16 state operated outpatient clinics. One of these clinics first became a research site for a CR randomized clinical trial, which allowed for a comparison of the feasibility and acceptability of CR in a research as compared to a clinical setting.

Results

The research site averaged almost triple the number of referrals as the clinical sites. Over nine months 46.51% of clinic referrals were enrolled in the CR program whereas 64.29% of research referrals were enrolled. Clinical site utilization averaged 70.53% while research site utilization averaged 90.47%. At the clinical sites, 97% of respondents reported CR was an excellent or good experience. There was high treatment fidelity for program structure and content across sites.

Conclusions

This comparison of CR in clinical and research sites highlights the decrease in referrals, enrollment and utilization that occurs when a program moves from a highly controlled setting to the real world. Still, the acceptability, fill rates and utilization indicated that CR can be successfully implemented in large scale, geographically diverse, publically funded clinic settings.

1. Introduction

Schizophrenia is associated with persistent neurocognitive deficits that do not respond to pharmacologic treatment and add to illness burden by imposing significant functional limitations. Cognitive dysfunction has been associated with the ability to adhere to pharmacotherapy for medical and psychiatric symptoms, to participate in and benefit from psychosocial programs, and to function in the social, vocational, and educational venues integral to community integration (Bowie & Harvey, 2006; Green et al., 2000; Green et al,. 2004; Kurtz, 2005; McGurk & Mueser, 2004).

Cognitive Remediation (CR) is an evidenced based behavioral training intervention that harnesses the potential for neuroplasticity that exists even when psychiatric illness negatively impacts brain functioning (Keshavan et al., 2014). CR improves neurocognitive processes (e.g., attention, memory, processing speed, executive function) and real-world functioning. Meta-analyses of CR randomized controlled trials indicate medium effect sizes (McGurk et al., 2007; Wykes et al., 2011) for people with schizophrenia spectrum disorders. CR typically involves certain core elements: time spent on restorative drill and practice exercises that target specific cognitive skills and clinician support to negotiate the tasks. Many programs also include discussion groups or simulation activities to facilitate learning transfer to real-world situations (Medalia & Bowie, 2016).

Although CR offers the promise of significant benefit with widespread adoption and implementation, to date it is largely unavailable in the United States. Numerous systemic factors may account for this: recognition of cognitive dysfunction and subsequent interventions are not a priority in behavioral health professional programs, and there are inadequate funding streams to support cognitive interventions (Medalia & Erlich, 2017). Another critical barrier is that CR requires unique conditions for implementation that many mental health clinics are ill prepared to provide, and this contributes to a gap between clinical science and practice. For example, CR requires a baseline assessment of cognition to determine need for treatment, and consistent consumer participation in a specified course of treatment with multiple sessions per week over several months. Restorative approaches typically require access to computer based drill and practice cognitive exercises and the Internet. Strategy-based approaches, which teach patients how to negotiate cognitive tasks, require treatment manuals and staff training. Many CR approaches rely on the availability of other psychosocial interventions like supportive employment or CBT to facilitate the transfer of learned cognitive skills to everyday life.

There is a need for research that addresses how to close the gap between the requirements of evidence-based CR and real-world conditions. To date, CR research has focused on internal validity (e.g., well-controlled efficacy trials) as opposed to ecological validity that would help extend findings to real-world contexts (Keshavan et al., 2014). Yet the achievement of more widespread and consistent implementation of evidence-based practices demands that we address external validity and local realities (Medalia & Erlich, 2017), which may impact the perceived contextual challenges associated with implementation. Understanding what happens to treatment fidelity in real-world contexts can inform strategies for adaptation and lead to a more comprehensive understanding of how to optimize effective treatments over time (Chambers & Norton, 2016).

This paper focuses on an initiative by New York State’s Office of Mental Health (OMH), with its academic partner Columbia University, to create the first known statewide level implementation of cognitive remediation programs for adults with a Serious Mental Illness (SMI) (Medalia & Erlich, 2017). The CR program, called Cognitive Remediation to Promote Recovery (CR2PR), was developed for patients with SMI identified as having cognitive impairments that hamper recovery goal attainment. It was initiated on adult outpatient services, with the goal of eventually becoming available at inpatient and forensic centers. Of the 16 adult outpatient clinics selected for implementation, one site requested to first be a research site for a cognitive remediation randomized clinical trial, before transitioning to a clinical site. This allowed for a comparison of the feasibility and acceptability of CR in a research setting as compared to clinical settings. The purpose of the study reported here was to evaluate the fit between the program conditions and treatment model in research and clinical settings, with the goal of informing future research on the contextual challenges associated with the implementation of CR.

2. Methods

2.1 CR Treatment Context

OMH is the largest state mental health system in the United States, and is relatively unique in the breadth of geographic regions, cultural and linguistic groups served. In addition to licensing mental health agencies across New York State, OMH manages state-operated services that provide inpatient and outpatient care, housing, and ancillary services to recipients in Psychiatric Centers (PC’s) for child, adult (age 18 and over), and forensic populations. As of the writing of this paper, OMH has 16 adult PC’s, which provide care to an estimated 19,387 unique adult consumers across 62 clinics.

The 16 clinics that participated in CR programming range in size, with the smallest serving 23 clients and the largest serving 299 per week. The mean weekly census across clinics was 138 clients. Between 92 and 100% of clients at these 16 clinics were diagnosed with a SMI. On average the percentage of patients carrying a diagnosis of schizophrenia or other psychotic disorders was 66.3% (range 40–87%). Adults age 18–64 years constituted over 80% of the population served in all but one clinic, where 60% of the clients were ages 65 and over. Ten clinics were in urban areas and six were in rural areas. One clinic served a majority (58%) of non-English speaking clients while the other 15 clinics served a majority of English speaking clients (range: 79–100%).

The CR2PR program recognizes that cognitive remediation’s impact on functional outcome is best realized when the treatment is tied to overall recovery goals (Amado & Sederer, 2016; Wykes et al., 2011). The three pillars of CR2PR are to maximize wellness, address cognitive deficits, and improve patient-centered recovery goals. Thus the CR2PR program was developed as an integrated component of individualized treatment planning, and was implemented in conjunction with other evidence-based and research-informed practices for SMI (e.g., Supported Employment/Education, Wellness Self Management).

2.2 Implementation Process

The CR2PR initiative required significant organization and administrative support coordinated through the Office of the Medical Director and the Office of State Operated Adult Services. During implementation, a leadership team worked to identify local and central leaders; find content experts and vendors; locate initial OMH state-operated adult outpatient sites for a staggered rollout of CR2PR; and ready facilities for the necessary staffing, training, and infrastructural requirements. A facility needs assessment of staff, space, and computers helped to identify the sixteen clinics that were ultimately chosen for geographic, socioeconomic, cultural and educational diversity and strong leadership factors. Other critical implementation elements involved identifying a neuropsychological assessment battery, a CR program that could be feasibly implemented within OMH, anticipating staffing and financial constraints, and adapting training materials to be specific to OMH’s SMI population. To facilitate dissemination of information for training and supervisory purposes, a website with CR2PR-related teaching and supervision materials was created on OMH’s internal website, and Columbia University’s annual CR conference (www.cognitive-remediation.org) and CR training website (www.teachrecovery.com) were made available to the CR2PR clinicians.

There were two implementation “phases” for the CR2PR program. In 2015, Phase I was implemented at nine OMH operated adult outpatient clinics that were attached to eight unique OMH operated Psychiatric Centers. In summer 2016, Phase II was launched with CR2PR implemented at seven additional OMH operated clinics whereby five new Psychiatric Centers received the program and two pre-existing Phase I Psychiatric Centers implemented CR2PR in a second clinic. By 2017, CR was available at 13 of the 16 Adult PC’s.

At each CR2PR site specially trained clinicians were designated as “champions” to oversee the program’s implementation, which included: coordinating neurocognitive screening and assessment; providing the CR intervention and working closely with the CR2PR directors to maintain fidelity to the program. One clinic chose to delay clinical implementation to become a site for a randomized clinical trial which offered two types of cognitive remediation (see clinical trials.gov NCT01945333), structured in the same format as the type of CR used in CR2PR clinics. Though conducted by research personnel, research-based CR groups were offered during regular clinic hours in the clinic’s designated space. In both the research and clinic programs, sessions were conducted in small groups for a total of 60 minutes, including 45 minutes devoted to restorative web based cognitive training exercises and about 15 minutes of bridging group discussion. Recommended group size was six. The web based training exercises used at the clinical and research sites were similar however the research study required three times a week attendance for 30 sessions while the clinical sites required twice a week attendance for a recommended 30 sessions. The research participants were paid $40 for completing the baseline and end of treatment assessments and $70 for the 3 month follow up assessment, whereas participants in CR2PR received no compensation for doing the only required assessment, which was at baseline.

2.3 Data collection

At the clinical CR2PR sites, program evaluation is an integral aspect of the service. This includes a quarterly report on referrals, intakes, utilization, and discharges, as well as continuous monitoring of outcomes. Clinician and administrator perceptions of the implementation process are discussed during supervision with author AM. Service data and supervision of clinicians are used to monitor fidelity. See Appendix 1 for the treatment fidelity scale that was adapted from the generic scale recommended for clinical and trial use (Medalia et al., 2017). Consistent with guidelines for program evaluation in clinical settings (Corrigan et al., 2000; Goodman et al., 2016), outcomes were measured using a client completed satisfaction scale (see Appendix 2) and a clinical care coordinator report on recovery progress (see Appendix 3), both completed after 30 sessions. We present here the results for Phase I of CR2PR implementation during the first five calendar quarters that started in the 4^th quarter of 2015. The clinician-completed report on recovery progress was added as an outcome in the 4th quarter of 2016, and collected over three months. Data from the research site were collected over the course of three calendar quarters. To compare equal time periods we also report data from the first three quarters of CR2PR. The program evaluation data from clinical sites was deemed exempt from IRB review because they were quality improvement activities implemented for the specific purpose of positively impacting clinical care. The data obtained for the research site was collected with approval and oversight provided by the New York State Psychiatric Institute IRB.

3. Results

3.1 Site Characteristics and Feasibility

One site designated to start CR2PR in Phase 1 did not open because the clinic moved. The remaining eight Phase 1 sites were in urban areas, as was the site where research was done. All sites offered rehabilitation services including individual support for employment, education and social goals, and symptom management, as well as access to and liaison with community support services. Average number (SD) of clients served weekly at the CR2PR sites was 143.63 (77.27) and 58 at the research site. Average % patients served weekly with schizophrenia and other psychotic disorders was 70.87 (range 54.1–87) for the eight CR2PR sites and 75.9 for the research site. Average percent of clients age 18–64 was 86.84 for the eight CR2PR sites and 86.2 for the research site. Average % clients identifying English as their primary language was 84.51 for the CR2PR sites and 98.3 for the research site. Average % (range) of clients with a SMI diagnosis was 98.68 (96.4–100) for the CR2PR sites and 100 for the research site. The clients at the research site did not differ significantly from those at the eight CR2PR sites in terms of the diagnoses, age or primary language parameters (all p>.4).

Two of the eight Phase I sites encountered space constraints making it difficult to assemble six clients, the recommended group size. Three sites had significant problems obtaining web connectivity that were resolved with Central Office intervention. Six of eight sites encountered difficulty accessing the web based cognitive exercises due to statewide government Internet protocols that blocked access to many outside websites. Website blocks at all six sites were removed with Central Office assistance, which required a waiver for the use of the websites on the clinicians’ computers. Three sites had delayed start dates as they waited to get computers to be delivered. At the inception of implementation, all sites had a complement of licensed staff identified to work on the project including at least one psychologist.

By project initiation the research site met feasibility criteria for space, web connectivity and access to cognitive programs, but needed to add two computers to accommodate a group of six.

3.2 Referrals, enrollment and participant characteristics

Referrals to the clinical sites during three/five quarters came from psychiatrists (12.4/11.24%) and other mental health clinicians (87.6/88.76%). They were higher in the first quarter of the project (Q4 2015), with downward stabilization over time (see Figure 1). Across the eight clinics, 47.75% of referrals were made in the first quarter. The average number of new referrals per quarter at a given clinical site was 5.38 over three calendar quarters and 4.45 over five quarters. For the research group, referrals came only from non-physician mental health clinicians. On average 14 referrals were made per quarter with identical rates in the first and second quarter and a lower rate in the third quarter as the study stopped recruiting.

Figure 1.

Referrals to CR2PR over five calendar quarters

Enrollment criteria varied between the clinical and research sites in two ways: 1. Consent to participate in research was required at the research site and 2. All diagnoses were acceptable at the clinical sites but inclusion criteria for research enrollment required a schizophrenia or schizoaffective diagnosis. Even with more stringent enrollment requirements the research site had a higher enrollment rate. Across CR2PR clinics 46.51/51.69% of the referrals were enrolled in the CR program at the three/five quarter marks whereas 71.43% of patients referred to the CR research study consented to participate in the study; and 64.29% of original referrals and 90% of those who consented met criteria for enrollment.

Participants in the eight CR2PR and one research program were men and women between the ages of 18–64, who carried a SMI diagnosis, had an estimated IQ > 70, no active substance dependence, and no neurologic illnesses known to cause cognitive impairment. At the time of referral they were psychiatrically stable enough to be in outpatient treatment, and were receiving services at the clinic rehabilitation program that offered psychiatric and rehabilitation oriented behavioral treatments designed to promote engagement in productive community based activities.

3.3 Participation and commitment

Utilization measured the percent of time CR2PR clients participated in a scheduled session. Across clinical sites utilization averaged 70.53% (range 62.2–78.3%) over three quarters and 72.04% (67.1–75.1%) over five quarters. For the research site, utilization averaged 90.47%. At the clinical sites 10.12/19.46% of enrolled clients were discharged from the group due to a lack of commitment at three and five quarters respectively, while at the research site 11.11% were discharged due to lack of commitment.

3.4 Treatment Fidelity

During the first quarter of the project (Q4 2015) the most frequent lapse in treatment fidelity concerned identification of rehabilitation goals. By the second quarter (Q1 2016) there was 100% treatment fidelity at each clinical site for all elements monitored: treatment structure and session organization, group composition, clinician competence, session activities, and learning transfer activities.

3.5 Satisfaction and recovery

Across the clinical sites 64 clients self-reported a high level of satisfaction with the service (see Figure 2), with 96.9% reporting it was an excellent or good experience (see Figure 3). Across the 8 sites the overall satisfaction levels ranged from 3.3 – 4. Most clients thought the CR group helped improve cognition: 54.7% answered “definitely” and 42.2% answered “I think so”. Only 3.1% of clients responded “didn’t think so”. About 90% thought the CR group helped them to deal more effectively with situations at home, school, work, or with friends: 48.4% thought it “helped a great deal” and 41.9% indicated it “helped somewhat”. Only 9.7% declared it “didn’t really help”. For the three questions, no client answered the lowest score indicating that CR2PR did not help them in any way.

Figure 2.

Satisfaction ratings for CR in clinical settings (N = 64)

Rating of 4 is most positive. Ratings come from 8 sites

Figure 3.

Satisfaction survey: Overall CR experience (N = 64)

A questionnaire completed by the clinical teams evaluated if clients evidenced more engagement in some aspects of their day-to-day life after the group. Across CR2PR sites, 28 clients were evaluated: 89% showed increased engagement in either the clinic or community; 50% increased engagement in both the clinic and community, (see Figures 4 and 5 for domains of engagement).

Figure 4.

Domains of improved engagement in outpatient clinic programs (N = 19*)

* 19 of 28 rated clients showed more engagement in their clinic program

Figure 5.

Domains of improved community engagement (N = 20*)

* 20 of 28 rated clients showed more engagement in their community

3.6 Staff turnover and clinic closures

Three of the 16 clinics closed since the beginning of the project. There was a range of CR2PR staff turnover at the clinics: one clinic lost 80% of CR2PR staff while another had no staff turnover. On average, CR2PR staff turnover was 25%, meaning that across the eight sites one out of four people trained as clinicians to run the CR2PR program terminated employment. By comparison, at the grant funded research CR program there was 50% CR research staff turnover; one of the two CR research clinicians terminated employment.

4. Discussion

Cognitive remediation is an emerging evidence based practice that addresses the neurocognitive deficits known to impact real-world functioning among individuals with schizophrenia spectrum disorders. CR is largely unavailable in the United States (Medalia & Erlich, 2017); one reason for this slow uptake in practice may be that CR implementation requires unique conditions that many clinics are ill prepared to provide. There is a need for research that addresses how to close the gap between the requirements of CR and real-world conditions needed to implement it. By understanding what happens to treatment fidelity in real-world contexts, we can develop strategies for adaptation and better optimize effective treatments over time (Chambers & Norton, 2016).

This paper describes the first known statewide program in the United States to address cognitive health in individuals with psychiatric illnesses. The CR program, called Cognitive Remediation to Promote Recovery (CR2PR), was initiated at adult outpatient services in New York, with 16 sites chosen for implementation. These were all state operated mental health clinics that served predominantly adult SMI populations. One of these sites was designated a research site for a cognitive remediation randomized clinical trial, which allowed for a comparison of the feasibility and acceptability of CR in a research protocol as compared to clinical protocol, given similar implementation contexts. The clients at these clinics were similar in terms of age, diagnoses and primary language spoken. Even recognizing the imbalance of one research compared to 8 clinical sites, the data collected allowed us to evaluate the fit between the program conditions and treatment model in research and clinical settings, with the goal of informing future clinical initiatives and research regarding the contextual challenges associated with the implementation of CR.

CR requires access to computers, the Internet to access specific websites, and as implemented in this project, a room for six clients to be seen in a group. Measures of site feasibility indicated that internet connectivity and web access to the cognitive training tasks were the most common barriers to implementation. Large systems of care typically require a centralized approach to internet safety, with resultant blocks on websites to avoid viruses. In turn, centralized administrative support is needed to facilitate removal of website restrictions so program implementation can proceed. Interestingly, staffing was the least problematic feasibility barrier in these clinics, despite high turnover rates. There was considerable staff enthusiasm for both the clinical and research projects and the greatest rate limiting factor in maintaining staffing was the lengthy onboarding process required by any large system of healthcare.

Maintaining fill rate is essential to any clinical service. Without referrals, and efficient intake processes, the service will dwindle. We found that non-physician mental health workers were most likely to make referrals at these outpatient clinics. Indeed over 80% of referrals came from non-physicians. The reasons for this imbalance are worthy of investigation since the psychiatrists are essential members of the clinical team, and potentially an important referral source. Understanding the avenues of communication between the psychiatrist and other mental health workers, appreciating workload constraints, addressing attitudes about cognitive dysfunction in SMI populations, and recognizing the efficacy of CR as a treatment are all factors to consider as limiting physician referrals.

In terms of the actual number of referrals to CR, the research site averaged almost triple the number of referrals as the clinical sites, even though it served far fewer clients than the average clinical site. Referral rates were highest at the clinical sites during the first quarter of program operation, when more staff resources were allocated to group start-up. Following treatment initiation, designated clinical time was divided between running CR groups, charting, and interfacing with the team and supervisor to generate and follow-up with referrals. On the other hand, the research site had a coordinator designated to encourage referrals, and benchmarks to monitor in order to meet funding requirements. This data indicates that sharp declines in referral rates can be expected when there is a transition from research to clinical programs. At OMH there was an educational program to encourage awareness about cognition and the CR program (Medalia & Erlich, 2017), but structuring of staff time to promote referrals may also be necessary.

Another component of fill rate is enrollment. From referral to enrollment is a staged process, and clients may exit that process for different reasons, such as not meeting eligibility criteria, lacking time and or commitment to the program. The research study had more stringent eligibility criteria that might have lowered the enrollment rates, but in fact enrollment in the research study was much higher. Whether considered from the standpoint of consent rate (71.43%) or meeting eligibility criteria (64%), enrollment at the research site was higher than the 46.51% enrollment achieved at the clinical sites in the same time period. While having a dedicated study coordinator likely contributed to this result, participant motivation may also be a factor impacting a client’s choice to participate in a research protocol versus a clinical service. This could have also played a role in the differing rates of utilization at the research and clinical sites.

Participation in sessions is a sign of the acceptability of the treatment. When a participant shows up to a scheduled session it means they have judged the cost benefit of attendance to be weighted toward benefit and acceptability. In neither the research nor clinical setting was there monetary gain associated with participation in scheduled sessions, although research participants did receive compensation for the pre and post treatment assessments. In the research setting, participants attended offered sessions (utilization) 90.47% of the time whereas in a similar period, utilization averaged 70.53% at the clinical sites. Both indicate acceptability, but the drop off at the clinical sites is noteworthy. As the rates of enrolled participants discharged from group due to lack of commitment were fairly similar for the research (11.11%) and clinical (10.12%) sites, it is unlikely that is a factor. In supervision, clinicians and supervisors consistently referenced clinic ethos and attitudes about participation when asked about client motivation to attend sessions. Clinicians were reluctant to place demands for more consistent, twice weekly attendance at the CR program, which deviated from the norm of requiring once weekly attendance common to other clinic groups. In contrast, at the research site, a coordinator and clinician urged attendance three times per week, and made concerted outreach efforts when sessions were missed. That, as well as other intrinsic and extrinsic motivators associated with participating in the research protocol may have impacted attendance. It is noteworthy that in fact clients found a three times a week research program quite acceptable, which contradicts assumptions about preferred treatments being infrequent.

Treatment fidelity refers to the strategies used to monitor and enhance the reliability and validity of behavioral interventions. In the CR2PR programs, maintaining treatment fidelity allowed for clarity about whether it is in fact the CR intervention that is the primary mechanism for changes in cognition and functioning. Four months into operation of the program, all clinical sites had 100% treatment fidelity for all monitored elements including treatment structure and session organization, group composition, clinician competence, session activities, and learning transfer activities. Regular supervision of the clinicians, who were active participants in program evaluation monitoring likely contributed to this high rate.

Outcomes evaluation in clinical settings requires the use of measures that are consistent with the goals of providing the least burdensome and least expensive care. Unlike research settings, where repeat cognitive testing is a funded integral aspect of efficacy monitoring, in clinical settings there is not funding to pay for repeated cognitive assessment. Client satisfaction ratings and feedback from the clinical team, which are recognized program evaluation tools (Corrigan et al., 2000; Goodman et al., 2016) were employed in this program. While neither are objective, such measures provide an important perspective on participant experience. With respect to this program, it was quite positive. Clients were overwhelmingly satisfied about the three aspects of CR2PR program queried: overall satisfaction, impact on cognition and its effect upon daily functioning. Clinicians also saw the majority of clients evaluated as evidencing more engagement in the clinic or community after 30 sessions. Increased engagement was most often evidenced as increased self-confidence and increased participation in community based activities. This is consistent with reported improvement in self - esteem and quality of life for people with schizophrenia who participate in CR (Garrido et al. 2016). Researchers may want to consider this as they consider ways to improve assessment of relevant functional outcomes. When functional outcome assessments focus exclusively on competencies (e.g., managing money, travel skills), they may miss the functional elements that clinicians and clients consider important.

The transition of treatments from research to community settings is known to be complicated and difficult. Despite best efforts to identify stable community clinic settings for implementation, within two years, three of the chosen CR sites closed, and there was 25% staff turnover across sites. Such changes are expected in any clinical program implementation, but the consequences remain unpredictable. However, despite these changes, the resilience of sites was remarkable. A site that lost 80% of its staff developed a thriving and well-accepted CR2PR program. Staff at sites that closed planned to open CR programs in their new setting. The authors believe that ongoing supervision of sites, empowerment of sites at a local level while providing strong central administration support, and a nimble, efficient staff training program were essential elements to the ability of the system to withstand the inevitable stresses that occur on implementation efforts. Importantly, the outcomes assessment indicated that the program itself was clearly responsive to client needs.

By comparing the program evaluation data of clinical sites to the feasibility data of the research site we have been able to highlight the decrease in referrals, enrollment and utilization that occurs when a program moves from a highly controlled setting to the real world. It is noteworthy that this occurred even when there was high treatment fidelity for program structure and content.

The New York State OMH CR2PR project is an example of a successful clinical implementation project: fill rates, acceptability and outcomes were high. Still there is always room for improvement. Further research will be needed to better understand why clients successfully attended three times a week CR at the research site but twice a week attendance at the clinical programs was considered burdensome. The use of web-based interventions in large systems of care is an emerging practice. While there is considerable enthusiasm for CR in the consumer community (Keshavan et al., 2014) sites are not always equipped to provide the necessary technical components for implementation. Therefore carefully designed programmatic structure that takes into account information technology needs is essential for large-scale implementation. More research to define the key elements of programmatic structure will facilitate future CR implementation efforts.

Table 1.

Fill and acceptability of cognitive remediation groups

	Clinical		Research
Time Frame*	3	5	3
Average number of referrals per quarter	5.38	4.45	14
% of referred clients enrolled	46.51	51.69	64.29
% utilization	70.53	72.04	90.47
% clients discharged due to lack of commitment	10.12	19.46	11.11

^*calendar quarters