Evaluating Training Programs for Primary Care Providers in Child/Adolescent Mental Health in Canada: A Systematic Review

Abstract

Introduction

The need for child/adolescent mental health care in Canada is growing. Primary care can play a key role in filling this gap, yet most providers feel they do not have adequate training. This paper reviews the Canadian literature on capacity building programs in child and adolescent psychiatry for primary care providers, to examine how these programs are being implemented and evaluated to contribute to evidence-based initiatives.

Methods

A systematic literature review of peer-reviewed published articles of capacity building initiatives in child/adolescent mental health care for primary care practitioners that have been implemented in Canada.

Results

Sixteen articles were identified that met inclusion criteria. Analysis revealed that capacity building initiatives in Canada are varied but rigorous evaluation methodology is lacking. Primary care providers welcome efforts to increase mental health care capacity and were satisfied with the implementation of most programs.

Discussion

Objective conclusions regarding the effectiveness of these programs to increase mental health care capacity is challenging given the evaluation methodology of these studies.

Conclusion

Rigorous evaluation methods are needed to make evidence-based decisions on ways forward to be able to build child/adolescent mental health care capacity in primary care. Outcome measures need to move beyond self-report to more objective measures, and should expand the measurement of patient outcomes to ensure that these initiative are indeed leading to improved care for families.

Introduction

It is estimated that 1.1 million Canadian children and youth under 20 years old suffer from mental health conditions that affect their daily lives, and mental health problems among children and youth are predicted to increase by 50% by the year 2020 (Canadian Paediatric Society, 2009). Child and adolescent mental health problems are the leading health problems that Canadian children face after infancy, and many adult mental health problems are believed to have their roots in the pre-adult years (Kelleher & Stevens, 2009). There is a continuing shortage of child psychiatrists and specialized mental health services (Lunsky, Garcon, Morin, Cobigo, & Bradley, 2007; Steele & Wolfe, 1999). Long-wait lists and service fragmentation, geographic location and transportation issues, financial barriers, and often sub-clinical symptoms and co-morbidity are significant barriers for families to accessing child and youth mental health care. It is estimated that about 75% of young Canadians with mental health concerns do not receive any specialized treatment (Waddell, 2007).

Primary care providers (PCPs) play a pivotal role in the recognition and management of child and adolescent mental health problems. It is estimated that about 75% of these issues are first seen in primary care settings, as PCPs tend to be the first medical professionals asked by children/youth and families to help manage behavioural or mental health problems (Evers-Szostak, 2000; Miller, 2007). PCPs are in a unique position to identify and manage children and adolescents with mental health issues for various reasons. Primary care is one of the most accessible and affordable health care settings for families to access, particularly for more rural and remote areas, and there is a lack of stigma associated with being treated by a PCP compared to a specialized mental health care provider (Sarvet, Gold, & Straus, 2011). The trusted and long-term relationship patients have with their PCP is important, and often the whole family will be patients of that provider, so the PCP is in a position to know the family dynamic and are able to provide family-centered care, as well as care for the whole child (mental and physical) (Foy, 2010; Hafting & Garlov, 2007; Sarvet et al., 2011). This long-term relationship with the child and family helps to facilitate early detection and identification of mental health concerns in the child as well as early parent-child problems (Fisman, Sangster, Steele, Stewart, & Rae-Grant, 1996). PCPs can also play an important role of acting as a resource and advocate for patients and families in a centralized place where they receive most of their care (Oandasan, Malik, Waters, & Lambert-Lanning, 2004).

Despite the important role of PCPs in the assessment and management of mental health issues in children and adolescents, a minority of PCPs receive adequate formal training (Fallucco, Hanson, & Glowinski, 2010; Fremont, Nastasi, & Newman, 2008; Steele et al., 2003; Steele, Lochrie, & Roberts, 2010; Steele et al., 2012). Most physicians report minimal undergraduate training in mental health, and there is minimal mental health training in residency programs (particularly focused on children and adolescents) (Steele et al., 2012). Results from a nationwide needs assessment regarding training in psychiatry with 847 rural/remote primary care providers across Canada has shown that these PCPs report lacking the confidence, skills, and adequate training to provide mental health care; however, they indicate that they recognize this gap in their training and would participate in programs to build their capacity to provide child/ adolescent mental health care if made available (Steele et al., 2012). This national needs assessment also indicated that these physicians want training on specific disorders such as Attention Deficit Hyperactivity Disorder (ADHD) and behavioural problems, anxiety, depression, and suicidality (which is understandable given these are some of the most common mental health concerns children and families present with). What is key is that these physicians want skills that they can use in their daily practice to manage these concerns, such as interviewing skills and therapeutic skills (Steele et al., 2012). These physicians also reported that they prefer these training programs to be delivered as Continuing Medical Education (CME) in the community, and in a small group format led by a family physician/child psychiatrist pair (Steele et al., 2012).

Primary care is a key forum to address the need for increased mental health care capacity. However, to guide current and future child/adolescent mental health care capacity building approaches we need a better understanding of previous capacity building efforts in child and adolescent mental health care for PCPs. Although there has been much international research in the area of pediatric mental health, an awareness is needed regarding the work being carried out in Canada. To our knowledge, no published review to date has focused on examining the Canadian research. Our aim was to review and summarize the published evaluation research on PCP capacity building programs in child and adolescent mental health in Canada through a systematic literature review. The main aims of this review were to: summarize the types of capacity building programs in child and adolescent mental health care for PCPs that have been evaluated; and, discuss the evaluation designs and methodologies used to be able to identify the gaps in the literature and then guide future research in this area.

Methods

The relevant articles were identified through searching the PubMed and PsycINFO databases. The following combination of key words was used: training, education, primary care, physician, mental health, psychiatry.

The inclusion criteria were as follows:

1. peer-reviewed articles published in English,

2. studies that occurred in Canada or with a Canadian sample,

3. studies in which primary care providers were provided training or education in child and adolescent mental health, and

4. studies in which some sort of empirical evaluation of the program was conducted.

Studies were excluded if they did not include an evaluation, if the program was only about adult mental health, or if the study was not conducted in Canada. There were no restrictions based on date of publication, and all types of evaluations (regardless of methodology) were included.

The initial search identified 465 articles. The abstracts were first screened in which 52 articles were identified as relevant for full-text review. Sixteen articles were retained for the systematic review that met the identified inclusion criteria (see Figure 1).

Figure 1.

Systematic review flowchart

These 16 articles were reviewed and the following information was extracted for the review: sample characteristics, location of study, content and format of the program, evaluation methodology, and evaluation results. A meta-analysis was not performed as various methods of evaluation were used across the studies, therefore a qualitative review was conducted.

Results

The sample characteristics of the 16 studies included in the review are presented in Table 1. The characteristics of the programs are described in Table 2.

Table 1.

Sample characteristics of the studies identified in the systematic literature review

Authors, year of publication	Location	Sample characteristics
Fisman et al., 1996	London, Ontario	33 family medicine residents in 5 groups
Garcia-Ortega et al., 2013	British Columbia	88 family physicians in 2 groups
Gaylord et al., 2015	Northwestern Ontario	5 primary care clinics and 5 outpatient community health centres.
Hunter et al., 2008	Ontario	Survey: 137 family physicians (mentees). Email correspondence: 26 family physicians (mentees) and 8 mental health psychiatrists (mentors)
Kates et al., 1998	Hamilton, Ontario	87 family physicians in 35 practices
Kirmayer et al., 2003	Montreal, Quebec	29 family physicians
Kisley & Chisholm, 2009	Halifax, Nova Scotia	4 primary care clinics
Kutcher et al., 2002	British Columbia	68 family physicians in 8 clinics across Canada (Vancouver, Toronto, Winnipeg, Halifax, Quebec, Ottawa, Thunder Bay, Calgary)
Kutcher et al., 2003	British Columbia	68 family physicians in 8 clinics across Canada (Vancouver, Toronto, Winnipeg, Halifax, Quebec, Ottawa, Thunder Bay, Calgary)
Naimer et al., 2012	Toronto, Ontario	43 family medicine residents and 4 psychiatry residents
Pignatiello et al., 2008	Toronto, Ontario	Primary care providers, 75 child psychiatrists, 100 patients.
Rockman et al., 2004	Ontario	100 family physicians, 10 psychiatrists, 10 general practice psychotherapists.
Spenser et al., 2009	Ottawa, Ontario	27 paediatricians, 16 outpatient mental health clinicians.
Stretch et al., 2009	Rural Southwestern Ontario	56 family physicians in 11 clinics
Swenson et al., 2008	Ottawa, Ontario	18 family practitioners and 22 family medicine residents. 256 patients aged 16 and older.
Vingilis et al., 2007	Hamilton, Ontario	79 family physicians, 39 mental health counsellors, 17 psychiatrists, and 8 registered dieticians across 36 primary care clinics.

Table 2.

Characteristics of the training programs from the studies identified for the systematic review

Authors, year of publication	Training program	Location and length of program	Program content
Fisman et al., 1996	Educational sessions	5 sessions from 1.5 to 2.5 hours delivered over a 6 month period at the primary care clinic where the physicians practice	Case presentation and group discussion, didactic lectures, and live interviews. Variety of topics driven by audience need.
Garcia-Ortega et al., 2013	Train-the trainer educational sessions	2 sessions delivered on October 2011 and January 2012 in Vancouver, BC.	Training PCPs in the Practice Support Program (PSP) to return to their communities and deliver the program locally. Program content focused on identification, diagnosis, and treatment of ADHD and Depression.
Gaylord et al., 2015	Shared mental health care program	Introduced in 2001, mental health professionals are located within the primary care clinic.	Family physicians and psychiatrists working collaboratively to deliver mental health services in the primary care clinic. Co-located mental health personnel only receive referrals from PCPs within their clinic.
Hunter et al., 2008	Mentoring program	Collaborative mental health care network program founded in 2001. Conducted via telephone/email.	Family physicians are connected with a psychiatrist mentor that they can contact via email or telephone when needed for timely advice and consultation.
Kates et al., 1998	Shared care program	Mental health counsellors work full time within the clinic and psychiatrists visit the clinic every 1–2 weeks.	Mental health counsellors and psychiatrists work collaboratively with the family physicians to increase mental health care capacity of clinic. Focus on integrated as opposed to parallel care.
Kirmayer et al., 2003	Cultural consultation program	Specialized mental health care service located within the local hospital.	Cultural consultation service for mental health available to primary care providers for referral for marginalized populations (immigrants, refugees, and ethnocultural minorities). Service provided specific cultural information, links to community resources, formal psychiatric assessments, and recommendations for treatment.
Kisley & Chisholm, 2009	Shared care program	Mental health professionals located within the primary care clinic to provide clinical support in direct care, consultation, liaison, and education.	Shared care mental health program for marginalized populations (i.e., homeless, living with addiction, living with disabilities, street youth, sole support parents, Aboriginal people, LGBT people, and racial minorities). Provided clinical support in direct care, consultation, liaison and education. Program also included outreach to shelters, temporary housing, and drop-in centres.
Kutcher et al., 2002	Educational program	Brief educational program (mixed lecture/seminar format) for treatment of depression. General program and enhanced program. Delivered on site at the clinic.	General program focused on epidemiology, neurobiology, clinical presentation, diagnosis, and treatment of depression and general use of citalopram. Enhanced program included this content and also focused more on specific strategies for improving recognition and treatment of depression, use of diagnostic/assessment tools, dosing of citalopram, and relapse prevention.
Kutcher et al., 2003	Educational program	Brief (60 minutes) educational program (mixed lecture/seminar format) for treatment of depression delivered on site at the clinic.	Subjects were epidemiology, neurobiology, clinical presentation, diagnosis, and treatment of depression in children and adults.
Naimer et al., 2012	Collaborative care program	Collaborative mental health care program between family medicine and psychiatry residents. Paired for one academic year.	Psychiatry residents paired (“buddied”) with family medicine residents. Family medicine residents contact buddy via email or phone for advice, consultation, etc. when needed.
Pignatiello et al., 2008	Telepsychiatry program	Program hub at SickKids Hospital in Toronto, with 15 other sites across Ontario.	Bilingual Telepsychiatry service for rural practitioners in Ontario to access consultations and education in underserviced areas.
Rockman et al., 2004	Mentoring program	Ontario College of Family Physicians (OCFP) mentoring program. Established in 2000, ongoing. Providers are all across Ontario.	10 groups with 10 family physicians, 1 psychiatrist, and 1 general practice psychotherapist in each. Family physicians can contact by phone or email those mental health professionals within their group for education and consultation.
Spenser et al., 2009	Collaborative care program	Shared care program (CHAT) developed by CHEO. Duration of 1 year. Located at the hospital’s community mental health centre in the urban core of Ottawa.	Program contained three parts: A paediatrician was integrated into the mental health care team to work part-time, clinicians were surveyed about their perceived educational needs, and continuing medical education (CME) in mental health for paediatricians and paediatric education sessions for mental health clinicians. CME sessions for mental health clinicians were seven hour-long talks given by paediatricians on: sleep problems, encopresis, child abuse, failure to thrive, adolescents with medically unexplained symptoms, and a “paediatric grab bag” of case vignettes. CME sessions for paediatricians was a single day-long event given by child/adolescent psychiatrists on developmental psychopharmacology, and paediatric anxiety and depression.
Stretch et al., 2009	Educational program	3.5 hour workshop on ADHD and Behaviour Disorders. Workshop held on site at 11 clinics in rural and underserviced areas.	Workshop included didactic teaching, observation of standardized videos demonstrating interviewing skills, and interactive group discussion. Workshop delivered by child psychiatrist and family physician. Topics focused on ADHD, conduct disorder, and oppositional defiant disorder, and interviewing skills for children/adolescents.
Swenson et al., 2008	Collaborative mental health care program	SHARE (Shared Mental Health Care Accessibility Research and Evaluation) program. September 2004 to December 2005. Located at two family practices in Ottawa.	Linked family practices with the mental health services at the local hospital. Mental health professional works in family clinic at least 1 day/week. The goals were to provide on-site mental health care to primary care patients, improve their access to hospital-based mental health services when needed, and improve access to primary care for mental health outpatients of the hospital who lacked a family physician. Adolescent and adult patients.
Vingilis et al., 2007	Shared care program	Hamilton Health Service Organization Mental Health and Nutrition Program. Implemented in 1994.	Integrate specialized mental health and nutrition services into family physician’s offices. 1 full-time mental health counsellor per 8000 patients, 1 part-time dietician per physician, 1 psychiatrist 0.5 days per month per physician. Goal to improve communication and collaboration between primary care and mental health professionals, increasing access to mental health care. For children and adults.

From this review, about half (seven out of 16) of programs are shared or collaborative care programs, in which co-located personnel (which includes a range of mental health professionals such as psychiatrists or mental health counsellors) are placed within the primary care clinic or office to increase access to mental health care by working with primary care providers. Primary care providers have within office access to faster referrals and can provide ongoing collaborative care (where both the family physician and the mental health professional are working on treatment planning and providing care), as well as easy access to informal educational opportunities and consultation by having access to mental health professionals within their office (e.g., Gaylord, Bailey, & Haggarty, 2015; Kates, Craven, Crustolo, & Nikoloau, 1998; Swenson et al., 2008; Vingilis et al., 2007). This review found there were two broad categories that shared or collaborative care programs fell into: A broad approach in which the goal is to increase the overall mental health care capacity of PCPs for the child and adolescent patients seen in their practice; and, a shared or collaborative care model that had a more specific focus on addressing the needs of marginalized and underserved populations (i.e., focusing on immigrant and refugee populations, LGBT people, homeless or street youth, etc.).

Also common were formal educational programs (five out of 16) specifically teaching on topics in child and adolescent mental health (which varied in their delivery from didactic teaching to case presentations and group discussion) for family physicians and family medicine residents, delivered often by a psychiatrist (e.g., Fisman et al., 1996; Garcia-Ortega et al., 2013; Kutcher et al., 2002; Stretch et al., 2009). As well, there was training in consultation and referral to specialized child and adolescent mental health care services, which involved linking primary care clinics with local specialized mental health care services to increase access to services, reduce wait times, and foster collaboration in the delivery of care between the primary care provider and mental health specialist. There were formal mentoring programs for PCPs in which they were paired with mental health professionals to assist with consultation and education (Hunter et al., 2008; Naimer et al., 20120; Rockman et al., 2004).

An examination of the topics covered in the formal education programs revealed depression as the most common disorder covered. One training program took a more general approach covering a range of topics that were driven by audience need with topics covering a variety of disorders (depression, anxiety, ADHD, eating disorders, suicide, learning disabilities, suicide, medication, interviewing skills, diagnosis, etc.) (Fisman et al., 1996). One program specifically focused on diagnosis, treatment, and medication for depression (Kutcher et al., 2002; 2003), another program focused specifically on diagnosis and treatment of depression and ADHD (Garcia-Ortega et al., 2013); and another program focused on interviewing skills, and diagnosis and treatment of ADHD and behavioural disorders (Stretch et al., 2009).

A summary of the evaluation methodology used in these 16 published studies and a summary of results are included in Table 3. Five studies utilized a post-program survey with no control group, four studies used a pre-post survey with no control group, and five additional studies used qualitative methodology (one study using qualitative methodology also used quantitative methods). One study employed a pre-post survey with a control group comparison, one study used a pre-post-follow up survey design without a control group comparison, and a final study employed a retrospective post-program chart review with no control group comparison. In total, one study was a mixed-methods methodology, with 11 studies employing quantitative methods, and four studies employing qualitative methods. Two of the studies had a control group for comparison, with one study collecting follow-up data.

Table 3.

Summary of evaluation design and measures from studies identified in the systematic literature review

Authors, year of publication	Evaluation Design	Evaluation Measures	Summary of results
Fisman et al., 1996	Post-training questionnaire. No control group	Likert scale items to indicate self-report confidence in ability (4-point scale) and self-report helpfulness of program (5-point scale) in regards to 17 identified knowledge, attitudes, and skills learning objectives.	76% response rate (n=25). Majority reported confidence in knowledge, attitudes, and skills. Majority of participants across the 5 groups reported sessions as helpful (69%–91%) with didactic sessions reported as least helpful (38%).
Garcia-Ortega et al., 2013	Baseline and post-training surveys. No control group.	Likert scale items to assess self-report confidence in using practice support tools provided in program and self-report confidence in mental health care delivery. 20-item true/false knowledge measure on child/youth mental health.	100% response rate at baseline, 47.4% response rate at post training for group 1, 64.3% response rate for post-training survey for group 2. Comparison from baseline to post-training shows an improvement in confidence using tools, confidence in providing mental health care, and increased knowledge of child/youth mental health.
Gaylord et al., 2015	Retrospective comparison of referrals from clinics with shared mental health care program to a (control) clinic without a shared care program.	Chart reviews to assess frequency of referrals to mental health services for depression, anxiety, psychosocial problems and psychotic symptoms.	Assessed 4600 referrals. On average the clinics with a shared care program made more referrals to mental health services compared to the clinic without a shared care program.
Hunter et al., 2008	Descriptive/qualitative study of mentoring network	Quality assurance survey conducted in 2004 with and qualitative analysis of email	From survey data: Satisfaction with program was high (88%). Most common requests for help are for nonemergency clinical issues, pharmacotherapy, psychotherapy, treatment review, and diagnosis. From email correspondence: Mentors convey information directly and indirectly. Trusted relationship between the mentor and mentee was important for effective communication.
Kates et al., 1998	Qualitative description	Participants’ self-report of benefits of the program for providers and patients after program implementation.	Benefits reported include support of family physician to provide timely mental health care, enhanced continuity of care, patients are able to receive more accessible mental health care at their family doctor’s office.
Kirmayer et al., 2003	Post-program surveys. Post-program chart reviews. No control group.	Provider self-report satisfaction with program. Chart reviews for first 100 of those referred to service consultations.	86% of providers reported satisfaction with the program. Useful aspects of program identified as increased cultural understanding, improved communication, and improved treatment. Chart reviews show that most common reasons for consultation was for clarification of diagnosis or symptoms (58%), request for help with treatment planning (45%), and requests for information related to a specific ethnocultural groups (25%).
Kisley & Chisholm, 2009	Pre- and post-program surveys.	Wait times for consultation, mental health functioning (based on DUKE scale), patient self-report satisfaction, self-report provider satisfaction, self-report provider knowledge, self-report provider confidence.	Median wait time for referral was 6 days for sites with program, 39.5 days for control site. 71–78% patients were satisfied with service. Improvement in general and mental health from pre to post in sites with program, but not at control site. Providers reported satisfaction with service and increased sense of knowledge and confidence from pre to post at program sites.
Kutcher et al., 2002	Pre- and post-program surveys. No control group.	Depression knowledge measure created for purposes of this study, chart reviews to assess diagnosis and pharmacotherapy. Self-report patient satisfaction.	Increase in knowledge from pre to post. Enhanced group more likely to use diagnostic tool, more likely to record depression diagnosis, and more likely to initially prescribe citalopram than general group. No difference between general and enhanced group in patient satisfaction, compliance, or treatment outcome.
Kutcher et al., 2003	Pre- and post-program surveys. No control group.	25 item yes/no depression knowledge measure created for the purposes of this study. Had two parts to assess general depression knowledge and depression treatment.	Participants had high baseline knowledge of depression. Significant increases in general depression knowledge and depression treatment knowledge among participants from pre to post.
Naimer et al., 2012	Post-program surveys only. No control group.	Self-report provider utilization of program, self-report provider perceptions of program usefulness, self-report provider satisfaction with program.	80% of family medicine residents used their psychiatry resident buddy during that academic year. In year 1, 93% of family medicine residents reported being satisfied, in year 2, 50% of family medicine residents reported being satisfied. All residents reported the program was useful.
Pignatiello et al., 2008	Qualitative evaluation methodology using surveys, focus groups, and interviews after program implementation. Case reviews of 100 consultations.	Self-report provider perspectives of service providers, family members, and consulting psychiatrists. Case reviews to assess outcomes of consultations.	Providers reported enhanced capacity to support clients, and increased confidence in diagnosis. Families reported reduced burden to travel for speciality care. Consulting psychiatrists reported valuing their role as consultants and were able to provide services without increasing ongoing caseload. Case reviews showed that multiple recommendations were made for youth with extremely complex needs. Successful implementation of recommendations was facilitated by specific treatment directives, availability of follow up sessions, and meaningful engagement of the young person and their family. Barriers were brief nature of consultation, and scarcity of community resources.
Rockman et al., 2004	Pre- and post-program surveys. Post-program chart reviews. No control group.	Baseline survey of providers assesses self-report access to specialized mental health care and self-report mental health care knowledge. Assessed self-report use of program and satisfaction every 3 months. Chart reviews to assess consultation patterns.	Survey results show that providers reported less difficulty accessing mental health care help, increased knowledge and confidence in providing mental health care, and the majority of providers were very satisfied with the program. Chart reviews indicated that most family physician visits were for depression and family dysfunction. Consultations were most frequently for bipolar disorder, schizophrenia, and addictions.
Spenser et al., 2009	Post-program surveys. No control group.	Questionnaires assessed provider self-report confidence and perceived effectiveness of program. Surveys to assess educational needs of paediatricians and mental health clinicians.	100% of providers reported satisfaction with having paediatrician on the mental health team. The top three medical problems posing challenges for mental health clinicians: sleep problems, seizures, and obesity. 83% of mental health clinicians reported satisfaction with CME sessions, 78% reported feeling more confident and knowledgeable. All paediatricians reported satisfaction with CME sessions.
Spenser et al., 2009	Post-program surveys. No control group.	Questionnaires assessed provider self-report confidence and perceived effectiveness of program. Surveys to assess educational needs of paediatricians and mental health clinicians.	100% of providers reported satisfaction with having paediatrician on the mental health team. The top three medical problems posing challenges for mental health clinicians: sleep problems, seizures, and obesity. 83% of mental health clinicians reported satisfaction with CME sessions, 78% reported feeling more confident and knowledgeable. All paediatricians reported satisfaction with CME sessions.
Stretch et al., 2009	No control group. Pre-, post- and 3 month follow-up questionnaires. One month follow up interviews.	Questionnaires assessed self-report provider confidence in managing ADHD and behaviour disorders. Self-report provider satisfaction and self-report provider perceptions of usefulness of program/impact on practice.	Pre-post training questionnaires showed an increase in self-report confidence in providing care (for ADHD more so than behaviour disorders). All participants reported being satisfied with the workshop. The discussions and didactic information were identified as key features of the workshop. Participants suggested a need for up to date written material, as well as local resources and collaborating with mental health network.
Swenson et al., 2008	Qualitative methodology using focus groups and interviews.	Focus groups and individual interviews after program implementation to assess provider and patient perceptions of the program.	Physicians reported the SHARE program useful. Physicians used program to help with medication, diagnosis, and development of treatment plans. Physicians reported consultations had an educational benefit, and that program facilitated a wide range of mental health services. 89% of patient reported the care they received as excellent or very good. 77% of patients were satisfied with the service. Patients liked the speed and flexibility of referrals, the communication between the family doctor and mental health team, and reported relief for finally getting treatment.
Vingilis et al., 2007	Mixed-method (quantitative and qualitative) design. Post-program implementation evaluation. No control group.	Quantitative data from patient database. Qualitative data from focus groups with practitioners to assess perceptions of practitioners.	Quantitative data: Over 12 month period (2002–2003): 3223 patients referred to mental health staff, 3431 patients referred to dieticians. Continuing care was provided by specialists and family physician. Most common problems were depression, marital problems, and anxiety. Most common treatments were individual counselling, Cognitive Behavioral Therapy (CBT), assessment/recommendations, and supportive therapy. Qualitative data: Collaboration and education opportunities were highlighted by providers, with time constraints being a barrier to care. Providers reported less burden on external services and better access to care for patients, and shorter wait times for referral.

In terms of the quantitative measures used in the evaluations, overwhelmingly the pattern was to use Likert scale items to assess providers’ self-report perceptions after program implementation. Seven of the 16 studies assessed increase in confidence by asking participants whether they felt their confidence had increased as a result of participation, rather than using a continuous, standardized scale that allows for assessing change over time (Fisman et al., 1996; Garcia-Ortega et al., 2013; Kisley & Chisholm, 2009; Spenser et al., 2009; Stretch et al., 2009). Increase in knowledge was measured the same way in two of the studies, in that participants were asked whether they felt that participation in the program led to an increase in their knowledge (Kisley & Chisholm, 2009; Rockman et al., 2004). Three studies used an objective knowledge measure or test on the topic(s) focused on in the program, and then measured change or increase from pre to post (Garcia-Ortega et al., 2013; Kutcher et al., 2002; 2003). The utility or impact of the program was most commonly assessed by asking participants whether they believed the program was useful or had an impact on their practice (five out of 16 studies) (Fisman et al., 1996; Naimer et al., 2012; Rockman et al., 2004; Spenser et al., 2009; Stretch et al., 2009).

Five of the 16 studies employed qualitative methodology using methods such as interviews and focus groups. Two of these studies conducted a process or descriptive evaluation assessing providers’ experience with the program (Swenson et al., 2008; Vingilis et al., 2007). With three of the studies using qualitative methods to explore the participants’ perceived benefits and impact of the program (Hunter et al., 2008; Kates et al., 1998; Pignatiello et al., 2008).

Patient outcomes were assessed in some way in seven of the studies. The most common method was to conduct chart reviews to assess referral/consultation patterns, and patient outcomes such as diagnosis and medication (Gaylord et al., 2015; Kirmayer et al., 2003; Pignatiello et al., 2008; Rockman et al., 2004; Vingilis et al., 2007). Another study used a mental health functioning scale to assess patient change from pre- to post and also assessed patients’ self-reported satisfaction with the program (Kisley & Chisholm, 2009). One other study employed qualitative methods to assess the patient experience with the program and its perceived benefits (Swenson et al., 2008).

Overall the results of these evaluations showed that providers are satisfied with the programs they received and reported that they perceived that it increased their confidence and their knowledge, and reported that they perceived the program to be useful and have an impact on their practice. Those programs that did assess change with quantitative pre-post measures found an increase in providers’ knowledge, confidence, and a reduction in referral wait times for patients (in collaborative or shared care programs).

Qualitative results provided more depth in terms of the perceived benefits and impacts the programs had on providers’ practices and what patients perceived as beneficial.

Discussion

Empirical evaluations of these programs is paramount in supporting decisions regarding efficacy and efficiency to then be able to meet the goals of increasing quality and access to mental health care for children/adolescents and their families. This review of capacity-building programs in Canada has revealed a gap in the use of rigorous, gold-standard methods of evaluation. Although the gold-standard for an outcome evaluation is a randomized-controlled trial (RCT) pre-post-follow up design with a control group, none of the evaluations used this design. One study did use a pre-post-follow up design but did not include a control group, and the two studies that did employ a control group for comparison did not have random assignment to conditions. RCT methodology is needed to make causal conclusions, and where not possible, a pre-post-follow up design with a control group for comparison (in the absence of random assignment) would be the most informative. This methodology would allow for comparisons of the effectiveness of a program above and beyond usual care or another intervention, and that is key to being able to make informed decisions on where limited resources should be directed to support the most effective programs and methods to build child/adolescent mental health care capacity.

It was also found that there was a particular reliance on self-report evaluation measures in which providers report after program implementation if they believe they are more confident, knowledgeable, and the impact on their practice or the utility of the program. Self-report perceptions from participants are an important piece in evaluations, but they are often not sufficient to draw conclusions from because they are subject to certain biases (like confirmation bias) and are not able to be objectively or observably measured. The inclusion of standardized and validated scales, and more objective and observational measures used to assess change from pre to post and in comparison to a control should be more consistently applied in evaluation studies of these programs.

A common theme that emerged from the review of evaluation methods was that participant confidence or their perceived increase in confidence in providing child/adolescent mental health care was often used as a proxy for change or increase in the providers’ skills. If participant confidence was high after the program, or if there was an increase in participant confidence from pre to post, this was often the basis for conclusions about the increase in mental health care skills. Pursuant to this, the review also identified that there was a dearth of training in therapeutic or counselling skills for primary care providers in these programs. The skills focused on in these training programs were often on screening, diagnostic, or treatment skills; in particular, pharmacotherapy, with one of the programs including content on interviewing skills. However, research conducted in the U.S. has indicated that a focus on relationship-building as well as brief, broad-based therapeutic skills may be useful in improving PCPs’ ability to provide child and adolescent mental health care for disorders such as depression, anxiety, and behaviour problems, and also enhancing patient outcomes such as mental health functioning and family functioning (Brown & Wissow, 2009; 2012; Wissow et al., 2008a/b). Bringing this focus on broad-based therapeutic and counselling skills into PCP training in Canada can be one significant way to address this gap and build mental health care capacity among PCPs.

When patient outcomes were included in the evaluation measures, it was often in the form of conducting chart reviews. Chart reviews are an important piece of data to objectively and quantitatively track referral and treatment patterns and other forms of health outcomes; however, the patient perspective as well as their mental health functioning above and beyond accurate diagnosis should be included in evaluations to gain a more comprehensive assessment of patient outcomes. It is important that these mental health care capacity-building interventions are meeting the mental health care needs of providers as well as the needs of patients. In order to meet the needs of both groups, evaluation designs need to measure outcomes for PCPs and for patients.

The papers included in this review span over 20 years, and insights can also be obtained by examining changes in focus of these capacity-building programs and the methodological approaches over time. One shift that can be seen over time in the papers included in this review is that the shared or collaborative care model of capacity-building becomes more common. These shared or collaborative care programs increasingly focus on having co-located mental health professionals in the primary care office to work with PCPs in providing child and adolescent mental health care. This pattern could be thought of as reflective of the trend in Canada in which the focus of primary care systems is not just on first-contact services, but on interprofessional health care teams, health promotion/disease prevention, and continuity of care (Hutchison, Levesque, Strumpf, & Coyle, 2011). In regards to methodological approaches, it was noted that more significant changes can be seen in the mid- to late-2000s, where the inclusion of patient outcomes as well as using multiple evaluation methods (i.e., surveys, interviews, focus groups, chart reviews) occurs. This could be said to indicate that this body of literature is moving in a promising direction in which methodology is more comprehensive and inclusive, and moves beyond more descriptive measures of PCP samples.

The 16 Canadian studies reviewed here indicate that there is a wide variety in the types of mental health care capacity-building initiatives and programs that are being implemented in Canada, particularly in employing shared or collaborative care models. The literature also reveals that there is a high demand among primary care providers to increase their capacity to provide mental health care to their young patients, and they report feeling very satisfied with the programs they receive (across all forms of training programs) and feel that these programs do have an impact on their practice. The evaluations of collaborative or shared care programs also demonstrate the eagerness of PCPs to work collaboratively with mental health specialists to provide the best possible care, and also demonstrates the benefits of different types of health care providers contributing to treatment planning and continued care of these young patients.

Conclusions

Canadian research on mental health care capacity-building programs for PCPs are limited, but existing studies indicate that there are a variety of initiatives being undertaken in Canada. The programs described in this review can be labeled as capacity-building as the content of these programs was aimed at improving the capacity of primary care to provide mental health care to children and adolescents. Even for shared or collaborative care programs that had mechanisms to increase access to timely referral or consultation with mental health specialists, the overall goal of these programs were not simply to improve access to existing systems of care. A strength of these shared or collaborative care programs was that they also focus on increasing the capacity of PCPs to provide mental health care. However, several gaps in the Canadian literature can be addressed in future work;

There is a significant need for more rigorous evaluation methodology and the implementation of objective, standardized assessments of provider knowledge and skills. Most consistent application of these rigorous evaluation methods will improve our ability to make comparisons across programs as well as causal conclusions. This is necessary to be able to make evidence-based decisions regarding the most effective capacity-building programs.

The inclusion of patient outcomes needs to be expanded beyond assessment of diagnosis and pharmacotherapy. Patient perspectives need to be included in evaluation designs to ensure that their needs are being met when implementing these programs for PCPs.

The Canadian programs that have been evaluated thus far (and where the evaluation was published in peer-reviewed journals) focus on formal education, shared or collaborative care models, and mentoring programs. More rigorous evaluation methodologies will help guide decisions regarding which types of programs are having the most significant impact across different types of health care provider and patient populations. Additionally, these mental health care capacity-building programs should always be based on the expressed needs of both practitioners (e.g., Steele et al., 2012) and families (e.g., McGrath et al., 2011), to ensure than an enhancement of PCP knowledge and skills in the area of child/adolescent mental health is indeed leading to improved care for families.