Skip to main content
Journal of the Canadian Academy of Child and Adolescent Psychiatry logoLink to Journal of the Canadian Academy of Child and Adolescent Psychiatry
. 2013 Nov;22(4):296–302.

Program Evaluation of a Child and Youth Mental Health Training Program for Family Physicians in British Columbia

Iliana Garcia-Ortega 1,, Helena Kadlec 2,3, Stan Kutcher 1, Marcus Hollander 3, Liza Kallstrom 4, Garey Mazowita 5,6
PMCID: PMC3825470  PMID: 24223049

Abstract

Introduction:

This brief report presents findings from the program evaluation of a portion of an educational program developed to support family physicians in improving their mental health care competencies in children and youth in British Columbia.

Method:

The Child and Youth Mental Health (CYMH) learning module is part of a broader initiative from the Practice Support Program (PSP) of the British Columbia Medical Association and was created specifically to assist family physicians in improving their competencies in the identification, diagnosis and delivery of best evidence-based treatments for children and youth exhibiting the most common mental disorders that can be effectively treated in most primary care practices.

Results:

The initial results from the program evaluation demonstrate a substantial improvement in family physicians’ knowledge of child and youth mental disorders and their self-rated clinical confidence in identifying and treating (both pharmacologically and psychotherapeutically) the most common child and youth mental disorders. Furthermore, because the training protocol involves a team-based approach which includes specialist physicians as well as school counsellors and human services providers, collaboration between primary practice and other providers is enhanced.

Conclusion:

The initial results encourage broader roll-out and further evaluation of this program on a wider scale.

Keywords: child and adolescent mental health, primary health care, child and adolescent mental disorders

Introduction

Mental disorders account for the largest proportion of the disease burden in young people globally (Patel, Flisher, Hetrick, & McGorry, 2007). On average, one in five individuals will experience a mental disorder during childhood and adolescence, with the majority of those demonstrating mild to moderately severe disorders. Early identification and access to appropriate diagnosis and best evidence-based treatment for young people requiring mental health care are particularly important, as mental disorders substantially interfere with the trajectory of development, leading to significant negative impacts on academic and vocational success, social, family and interpersonal functioning, an increased risk for substance misuse and earlier mortality due to suicide or comorbid medical disorders (Patel, et al., 2007). Ideally, access to care for mild to moderate mental disorders in children and youth should be widely available through the primary health care system, and provided by physicians and other primary care professionals, and fully integrated into usual health services (Kutcher & McLuckie, 2010; Kutcher, 2011; Servili, 2012). Advantages to this approach include: increased access to appropriate care; decreased stigma associated with receiving mental health care in stand-alone specialty mental health services; better opportunities for prevention and early identification of mental disorders; decompressing pressure on specialty mental health services, thus increasing access for those with more complex mental health care needs; and, integration of mental health care with all other aspects of health care, thus providing treatment to the “whole person” and entire family (Mickus, Colenda, & Hogan, 2002; Kutcher, 2011).

Mental health problems and mental disorders constitute a large part of general practitioners’ work with children and adolescents, for which they may receive insufficient training (Lester, Glasby, & Tylee, 2004; Kramer, 1998; Sayal, 2004). Indeed, surveys of primary care physicians have consistently noted a self-identified need for the development of additional competencies to assist them in better addressing the mental health care needs of young people (Kerwick, Jones, Mann, & Goldberg, 1997). Effective interventions for treating the most common mental disorders in children and youth are now available and many should be able to be effectively implemented in primary care settings, when appropriate training, access to specialty consultation and availability of responsive referral systems are in place (World Health Organization & World Organization of Family Doctors, 2008; World Health Organization, 2008; Leslie, Weckerly, Plemmons, Landverk, & Eastman, 2004).

Integrating mental health care into primary care is not a new concept. Internationally, the Declaration of Alma Alta in 1978 identified the necessity to address most health care needs through primary care. More recently in 2008, the World Health Organization (WHO) and the World Organization of Family Doctors (WONCA) have collaborated on the development of an approach addressing this need. The recently developed “Zero Draft” of the Global Mental Health Action Plan (WHO, August 27, 2012 version) further underscores the need for integration, not co-location, of mental health care in general care. Canadian initiatives to enhance primary mental health care capacity include the Canadian Collaborative Mental Health Initiative nationally and provincially, and the Practice Support Program (PSP) in British Columbia, among others. These initiatives notwithstanding, current capacity to meet child and youth mental health care needs in primary care across Canada is limited, and the urgency to address this is increasingly recognized (Kutcher & McLuckie, 2010; Kutcher, 2011).

Primary health care practitioners are often faced with addressing child and youth mental health needs (Ani & Garralda, 2005; Kerwick, et al., 1997; Kramer, 1998). The mild to moderate variants of the three most common mental disorders that manifest during this developmental period, and which are amenable to effective intervention by non-specialty services, are anxiety disorders, attention deficit hyperactivity disorder (ADHD) and depression. They all have clearly defined diagnostic criteria, well established treatments (both biological and psychosocial), appropriate treatment monitoring tools, and clearly specified approaches to care easily accessible in various treatment guidelines and algorithms (for example: American Academy of Child and Adolescent Psychiatry; American Academy of Family Practice; Canadian Paediatric Association; Canadian ADHD Resource Alliance (CADDRA)).

However, insufficient attention has been paid to how these well-established approaches could be most parsimoniously implemented in primary health care by busy primary care providers. One strategy is to effectively enhance the competencies of primary health care providers to identify, diagnose, treat, manage and appropriately refer (if necessary) children and youths with mild to moderate versions of these three common disorders. This requires the development, delivery and evaluation of contextualized training programs designed to enhance capacity within primary health care by building mental health care capacity of primary health care providers that is consistent with their roles and scope of practice. Furthermore, this training should facilitate the integration of schools (where most young people are), community-based human services providers (including specialty mental health services) and existing primary health care providers, in order to create and maintain a seamless pathway to mental health care for young people (Kutcher & Wei, 2012; Wei, Kutcher, & Szumilas, 2011; Zachrisson, Rödje, & Mykletun, 2006).

In 2007, the PSP was established as a joint initiative of the Ministry of Health Services and the British Columbia Medical Association (BCMA), to address several gaps in primary care in British Columbia. To date, the PSP has developed, implemented and delivered a number of educational training programs to family physicians and their staff on several topics ranging from office management and efficiencies to a number of clinical areas, such as chronic disease management and adult mental health (http://www.gpscbc.ca/psp/practice-support-program). To help address the identified gap in child and adolescent mental health care provision, the PSP developers, in collaboration with the Sun Life Financial Chair in Adolescent Mental Health team (IWK Health Centre/Dalhousie University), created the Child and Youth Mental Health (CYMH) training program, to improve primary care providers’ competencies in the identification, diagnosis and delivery of best evidence-based treatments for children and youths exhibiting common mental disorders in primary health care settings. Additionally, the training program provided a framework for integrating human services providers across organizations/institutions (such as schools, community services, etc.), by inviting members of the young patient’s ‘wrap-around care team,’ where available, to participate in the training module (the potential members included: pediatricians, family practitioners and staff, child psychiatrists, school counselors, nurses, psychologists, community child and youth mental health clinicians, and youth outreach workers). Input from non-governmental organizations, such as parent mental health advocates (the FORCE) and the Patient Voices Network in BC, was also solicited.

An advisory panel comprised of representatives from the Ministries of Health, Education, Child and Family Development, the University of British Columbia, the Vancouver School Board, BC Psychologists Association, BC Children’s Hospital, and the BC Pediatric Society, among others, provided oversight to the development of this project which was managed by the staff of the PSP. A modular education program, based on best available evidence contextualized to real life application in primary health care, was created by a group of nationally recognized experts in child and youth mental health and primary care lead by the Sun Life Financial Chair in Adolescent Mental Health knowledge translation team. Six knowledge components were created: Life span; Identification, Diagnosis and Treatment of Child Anxiety; Identification, Diagnosis and Treatment of Adolescent Anxiety; Identification, Diagnosis and Treatment of Child ADHD; Identification, Diagnosis and Treatment of Adolescent ADHD; Identification, Diagnosis and Treatment of Adolescent Depression. Each module contained practice relevant information including various clinical tool options to be used as screening, diagnostic and treatment management aids. These were all collated into a practice support toolkit. All components and the practice support toolkit underwent extensive external review by members of the advisory panel and other health, primary care and mental health experts at the request of the panel, with revisions applied based on reviewers’ recommendations.

In addition, a training program package consisting of slide decks and trainers’ notes supporting each educational module was created. The practice support toolkit was further enhanced with “how to use” information about each of the practice support tools following the process described above.

Communities and primary care sites across British Columbia were contacted by the PSP program staff to determine interest in participating in the program. Program support was provided to those groups who brought together primary care physicians, other primary health care providers and representatives of schools and community based human services providers to participate in co-training sessions. A train the trainer approach was utilized in which trainer teams consisting of family physicians, other primary health care providers, human services providers and school personnel from each participating community were trained. Training sessions were provided by content experts who had participated in the development of the modules and training program as well as representatives of parent advocacy organizations (the FORCE and the Kelty Centre) and youth who have experienced a mental disorder. Training program participants returned to their communities to deliver the training programs locally. PSP program staff provided ongoing support. An independent evaluation of the training sessions and community-based outcomes of training application was carried out by Hollander Analytical Services Ltd (HAS-L; the full report (lead author Helena Kadlec PhD), can be found at: www.practicesupport.bc.ca/psp/practice-support-program).

This brief report presents a summary from the program evaluation pertaining to the experiences of the family physicians who participated in the training program. The findings are based on a series of surveys created by HAS-L which were completed by the participants at two “Train-the-Trainers” (TtT) sessions that took place in Vancouver, BC, on October 4–5, 2011 (TtT-1) and January 25, 2012 (TtT-2) (Kadlec & Hollander, 2011; Kadlec & Hollander, 2012).

Family Physician Participants

Family physician participants in each of the sessions completed surveys unique to each training session. Thirty-eight family doctors participated in TtT-1 and 42 in TtT-2. Response rates to surveys were 100% at baseline (completed at the beginning of TtT1) and 47.4% at the end of TTT1, and 64.3% at the end of TTT2, respectively (Kadlec & Hollander, 2011; Kadlec & Hollander, 2012).

Findings from the Program Evaluation

Confidence in Using the Practice Support Tools for Assessment

During the training sessions, family physicians were introduced to various practice support tools (see Table 1). The physicians were instructed in the purpose and methods of the use of each tool. Self-reported comfort with using these tools improved substantially as a result of the training sessions, both as a function of the initial exposure (compare baseline with TtT-1) as well as over the course of the action period (compare TtT-1 with TtT-2).

Table 1.

Family physician’s confidence with the use of practice support tools

Tool / Resources Familiar with tools / resources at baseline (N=36) % At least “moderately” confident at end of TTT-1 (N=17) % At least “moderately” confident at end of TTT-2 (N=22) %
Risk Identification Profiles N/A 88 93
Psychotherapeutic Support for Teens (PST) 15.2 77 92
Teen Functional Assessment (TeFA) 22.6 88 100
Tool for Assessment of Suicide Risk in Adolescents (TASR-A) 18.2 94 93
Clinical Global Impression–Improvement Scale (CGI) 24.2 94 100
CRAFFT – Substance Use Assessment 18.2 71 62
Kutcher Adolescent Depression Scale (KADS-6) 21.2 94 95
Swanson, Nolan, and Pelham-IV (SNAP-IV) – 18 items 21.2 77 95
Screen for Child Anxiety Related Disorders (SCARED) 24.2 88 100

Knowledge Acquisition

Changes in knowledge about child and youth mental health were measured using a 20 item true/false test delivered immediately pre- and post-program. The average post-test scores (overall percent correct, averaged across physicians) were significantly improved over pre-test scores (independent-samples t (53) = 4.96, p<.001, Cohen’s d 1.41), demonstrating substantive improvement in knowledge as a result of the program. Table 2 shows the percent of physicians who correctly answered each of the 20 items.

Table 2.

Results on child and youth mental health the knowledge test

Test Item [correct answer with explanation] GPs with correct answer at TTT-1 (N=36) % GPs with correct answer at TTT-2 (N=19) %
1. Of the three most common mental health disorders among youth (anxiety, ADHD and depression), ADHD is the most prevalent. [F – anxiety is] 50.0 78.9
2. The onset of most mental health disorders occurs prior to age 25. [T] 86.1 100
3. Both ADHD and Anxiety Disorders have a strong genetic component. [T] 75.0 100
4. The best way to help a young patient suffering from a mental health disorder is to treat/help the family first. [F] 13.9 36.8
5. It is always necessary to involve the parents or guardians before treating a young patient, even if the young patient does not want his/her family involved. [F] 61.1 79.0
6. A marked change in emotions, behaviour, cognition or functioning, as self-reported by the youth and/or his/her parents, is a high risk factor for a mental health disorder. [T] 94.4 100
7. If a youth presents with mental health concerns and there is high risk for possible adolescent depression, this should be followed up with some form of contact (e.g., phone, email, text) within 7 to 10 days. [F – within 3 days] 0 5.3
8. If a young patient mentions a music band that you have not heard of, a good way of building rapport with the young patient is to act like you are familiar with it. [F] 94.4 100
9. Once the patient’s symptoms and function improve substantially, it is sufficient to repeat a “standing mental health check-up” every 3–6 months. [T] 50.0 57.9
10. Psychotherapeutic support for adolescents includes providing them with education about the illness and the treatment. [T] 100 100
11. The mood enhancing prescription (MEP) includes medication(s) to help a depressed adolescent feel better. [F] 33.3 63.2
12. The “worry reducing prescription” (WRP) is a general tool that is helpful in managing stress. [T] 66.6 89.5
13. If an uncontrollable issue regarding the child’s relationship with a parent arises during a visit, a useful strategy is to seek advice from peers or a mental health consultant who is/are familiar with that issue. [T] 83.3 89.5
14. It is a common misconception among lay persons that psychiatric medications used to treat mental disorders are not usually effective. [T] 63.9 78.9
15. One common symptom of childhood anxiety related disorders is worrying excessively about being as good as other kids at school. [T] 97.2 100
16. One common symptom of adolescent social anxiety disorder is worrying excessively about going to a party with a bunch of friends. [F – going alone, not with friends] 11.1 0
17. Induction of hypomania is often a side effect of medication treatment for ADHD. [F – rare, but happens] 55.6 84.2
18. Maternal depression is a risk factor for MDD in adolescents. [T] 88.9 100
19. Once a major depressive disorder is suspected in an adolescent, it is appropriate to begin medication immediately. [F – 3 weeks after PST] 80.6 73.7
20. If a young patient has been successfully treated in your office but then suffers a second episode of depression, it is time to refer him/her to a psychiatrist. [T] 41.7 47.4

Self-reported Confidence with Mental Health Care Provision for Children and Youth

Participants were asked to rate their confidence in a variety of domains pertaining to their provision of child and youth mental health care. In most of the domains assessed, the percentage of family physicians who rated their confidence as at least moderate rose substantially as a result of the training program (Table 3).

Table 3.

Number (percent*) of family physicians who rated themselves as “very” or “moderately confident” in various domains related to mental health for children and youth: pre- and post-training comparison

How confident are you in your ability to perform each of the following activities at this time? Baseline (N=36) TTT-2 (N=20)
Identify young patients who may benefit from mental health care.
  Child 22 (61.1%) 19 (100%)
  Adolescent 30 (85.7%)
Build rapport with new young patients.
  Child 33 (91.7%) 19 (100%)
  Adolescent 32 (91.4%)
Initiate a conversation about mental health care with a young patient (and his/her family, as appropriate). 33 (91.7%) 19 (100%)
Provide guidance/information to improve the mental health of your young patients. 22 (61.1%) 19 (100%)
Assess (wording on the baseline survey)/Screen (wording on the TtT2 survey) the following in children and youth:
  Depression 21 (58.3%) 18 (94.7%)
  ADHD 14 (38.9%) 19 (100%)
  Anxiety 23 (63.9%) 19 (100%)
Diagnose the following in children and youth:
  Depression (not asked) 19 (100%)
  ADHD 18 (94.7%)
  Anxiety. 19 (100%)
Develop an action plan for your young patients requiring mental health care. 13 (36.1%) 19 (100%)
Treat common mental health disorders in children and youth using medications. 13 (36.1%) 16 (80.0%)
Monitor side effects of medications. 23 (63.9%) 16 (80.0%)
Treat common mental health disorders in children and youth using other protocol-driven interventions. 11 (30.6%) 19 (95.0 %)
Monitor common mental disorders in children and youth. 14 (40.0%) 19 (95.0 %)
Communicate your patient’s needs to other mental health care providers, as appropriate. 33 (88.9%) 20 (100%)
Communicate your patient’s needs to community partners (e.g., school counsellors), as appropriate. 20 (55.6%) 20 (100%)
Collaborate with community partners (e.g., school counsellors) in the mental health care of children and youth. 22 (61.1%) 19 (95.0 %)
Refer the young patients to specialists in the community, as appropriate. 31 (86.1%) 19 (95.0 %)
Provide culturally-sensitive guidance to patients requiring mental health care. 16 (44.4%) 19 (95.0 %)
*

Note: Percent change per item is calculated out of the total number of respondents per item

Limitations, Conclusions and Discussion

The above reported analyses were conducted as part of a descriptive program evaluation and not as a research investigation using a randomized controlled trial methodology. This limitation notwithstanding, the findings reported above demonstrate substantive positive improvement in family physicians’ knowledge of child and youth mental health and their self-rated clinical confidence in addressing common child and youth mental disorders as a direct result of participating in this PSP learning module. The training program taught primary care physicians how to use a variety of practice support tools and materials that were developed to help enhance their competencies in the identification, diagnosis, treatment and management of common mental disorders of children and youth in the primary care setting. The educational materials were developed by experts in the field of child and youth mental health and primary care (family physicians and pediatricians) and critically reviewed by independent experts and a multi-stakeholder advisory committee that included physicians, administrators, health and human services providers and parent advocates. This extensive, cross-sectorial approach to the development of the materials helped ensure that they contained best evidence-based information, were comprehensive, and were contextualized to the needs of patients, their families and primary health care providers.

A limitation of this report arises from the training context in which primary care physicians were exposed to the materials alongside other primary health care providers. Although we designed the educational experience to enhance cross-provider training, the program agenda was such that activities less important to primary care physicians were scheduled at the end of the session. As a result, many physicians did not remain until the agenda was completed, resulting in fewer physicians completing the assessments (47% in the first session and 64% in the second session) than desired. Modifications to future educational programs to address this issue will be made. Of interest, however, is that results obtained from session one participants are not substantively different than those obtained from session two participants despite the difference in completion rates.

Another limitation is that we report on physician self-ratings of confidence in their competencies rather than applying an independent evaluation of their competencies. We recognize that this additional metric will be important to measure in future evaluations of the impact of this intervention, as well as including other metrics and an appropriate research methodology for determining if this intervention will demonstrate improved patient care in the real world. Although these adaptations are currently being developed for future application, this report has demonstrated positive results in the first component of the measurement cascade, described above, demonstrating positive impact of a first step in this ongoing evaluative process.

We recognize that the effectiveness of this training approach in improving access to mental health care and outcomes of mental health care for children and youth cannot be determined from this evaluation, nor from naturalistic studies. Rather, rigorous randomized controlled trials and health systems analytics are needed before we can comfortably endorse the widespread dissemination of this approach. Additionally, we recognize that this training model may be enhanced by some of the approaches provided by more traditional collaborative or shared care models that use specialty consultation to primary care providers. Future assessments should consider evaluation of such blended approaches as well.

These preliminary results, if supported by further ongoing evaluations, suggest that this program and this training approach may hold promise for improving the ability of young people and their families to access much needed care for common mental disorders through their primary care providers. Currently, the program has been accredited by both Family Practice (Canadian College of Family Practice) and Specialty (Royal College of Physicians and Surgeons) bodies for Mainpro C and Maincert section 1 credit, respectively. Should this approach yield positive outcomes across different regions in Canada, and with methodologically different evaluations, it could provide a nationally applicable framework within a wider context of a “Pathways to Care Model” (Wei et al., 2011) that would contribute to improved mental health care for children and youth nationwide.

Acknowledgements/Conflicts of Interest

The authors have no financial relationships to disclose.

References

  1. Ani C, Garralda E. Developing primary mental healthcare for children and adolescents. Current Opinion in Psychiatry. 2005;18:440–444. doi: 10.1097/01.yco.0000172065.86272.c7. [DOI] [PubMed] [Google Scholar]
  2. Kadlec H, Hollander M. Report: Survey participants at the first “Train-The-Trainers” (TTT-1) Session in Vancouver, BC, December 9, 2010. 2011. Victoria, BC: Hollander Analytical Services Ltd; 2011. [Google Scholar]
  3. Kadlec H, Hollander M. Final report: Evaluation of the CYMH “Train-The-Trainers” (TTT) Process. 2012. Victoria, BC: Hollander Analytical Services Ltd; 2012. [Google Scholar]
  4. Kerwick S, Jones R, Mann A, Goldberg D. Mental health care training priorities in general practice. The British Journal of General Practice. 1997;47(417):225–227. [PMC free article] [PubMed] [Google Scholar]
  5. Kramer TGM. Psychiatric disorders in adolescents in primary care. British Journal of Psychiatry. 1998;173:508–513. doi: 10.1192/bjp.173.6.508. [DOI] [PubMed] [Google Scholar]
  6. Kutcher S, Wei Y. Mental health and the school environment: Secondary schools, promotion and pathways to care. Current Opinion in Psychiatry. 2012;25(4):311–316. doi: 10.1097/YCO.0b013e3283543976. [DOI] [PubMed] [Google Scholar]
  7. Kutcher S. Facing the challenge of care for child and youth mental health in Canada: A critical commentary, five suggestions for change and a call to action. Healthcare Quarterly. 2011;14(2):14–21. doi: 10.12927/hcq.2011.22359. [DOI] [PubMed] [Google Scholar]
  8. Kutcher S, McLuckie A. Evergreen: A child and youth mental health framework for Canada. Calgary, AB: Mental Health Commission of Canada; 2010. [Google Scholar]
  9. Leslie LK, Weckerly J, Plemmons D, Landsverk J, Eastman S. Implementing the American Academy of Pediatrics attention-deficit/hyperactivity disorderdiagnostic guidelines in primary care settings. Pediatrics. 2004;114(1):129–140. doi: 10.1542/peds.114.1.129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Lester H, Glasby J, Tylee A. Integrated primary mental health care: Threat or opportunity in the new NHS? British Journal of General Practice. 2004;54:285–291. [PMC free article] [PubMed] [Google Scholar]
  11. Mickus M, Colenda CC, Hogan AJ. Knowledge of mental health benefits and preferences for type of mental health providers among the general public. Psychiatric Services. 2000;51(2):199–202. doi: 10.1176/appi.ps.51.2.199. [DOI] [PubMed] [Google Scholar]
  12. Patel V, Flisher AJ, Hetrick S, McGorry P. Mental health of young people: A global public-health challenge. Lancet. 2007;14(369):1302–1313. doi: 10.1016/S0140-6736(07)60368-7. [DOI] [PubMed] [Google Scholar]
  13. Sayal KTE. Detection of child mental health disorders by general practitioners. British Journal of General Practice. 2004;54:348–352. [PMC free article] [PubMed] [Google Scholar]
  14. Servili C. An international perspective on youth mental health: The role of primary health care and collaborative care models. Journal of the Academy of Child and Adolescent Psychiatry. 2012;21(2):127–129. [PMC free article] [PubMed] [Google Scholar]
  15. World Health Organization . WHO Library Cataloguing-in-Publication Data. 2008. Mental health gap action programme: Scaling up care for mental, neurological and substance-use disorders; pp. 1–44. [PubMed] [Google Scholar]
  16. World Health Organization & World Organization of Family Doctors (Wonca) Integrating mental health into primary care: A global perspective. Geneva, Switzerland: WHO Press; 2008. [Google Scholar]
  17. Wei Y, Kutcher S, Szumilas M. Comprehensive school mental health: An integrated “School Based Pathway to Care” model for Canadian secondary schools. McGill Journal of Education. 2011;46(2):213–229. [Google Scholar]
  18. Zachrisson H, Rödje K, Mykletun A. Utilization of health services in relation to mental health problems in adolescents: A population based survey. BMC Public Health. 2006;6:6–34. doi: 10.1186/1471-2458-6-34. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of the Canadian Academy of Child and Adolescent Psychiatry are provided here courtesy of Canadian Academy of Child and Adolescent Psychiatry

RESOURCES