Effectiveness of simulation in psychiatry for initial and continuing training of healthcare professionals: protocol for a systematic review

Abstract

Introduction

Although most healthcare professionals must deal with patients with mental illness, many are not prepared for the various situations that can ensue. Simulation may be a powerful pedagogical tool for simultaneously teaching knowledge, skills and attitudes. We aim to assess the effectiveness of simulation for initial and continuous training in psychiatry for healthcare professionals.

Methods and analysis

A comprehensive search for randomised and non-randomised controlled studies and single-group pretest/post-test reports will be conducted in electronic databases including MEDLINE, EMBASE, Scopus, CINAHL, PsychINFO, ERIC, the Cochrane Central Register of Controlled Trials (CENTRAL) and the Web of Science (Science and Social Sciences Citation Index), with a detailed query. The reference lists of selected studies, key journals and trial registers will also be searched for additional studies. Two independent reviewers, following predefined inclusion criteria, will screen titles and abstracts first and then the full texts of the remaining articles. A third author will evaluate discrepancies to reach a consensus. It will include randomised controlled trial (RCT), non-RCT, pre-test/post-test design studies, post-test design for satisfaction evaluation and qualitative studies. Risk of bias will be assessed by using the Cochrane Collaboration Tool for assessing risk of bias in RCTs. Meta-analyses will be performed if we find sufficient studies that assess predefined outcomes and if their characteristics are not too different. The quality of evidence will be assessed by the Grading of Recommendations Assessment, Development and Evaluation. A narrative synthesis will be performed for qualitative studies and when meta-analyses are deemed not possible.

Ethics and dissemination

Ethics permission is not required. Dissemination will be through publication in peer-reviewed journals, national and international conferences, and the lead author’s doctoral dissertation.

Trial registration number

CRD42017078779.

Strengths and limitations of this study.

• This will be the largest and most comprehensive systematic review in the field to include all types of simulation in psychiatry.

• This review will follow the Cochrane Handbook for Systematic Reviews of Interventions and will be reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis statement.

• Both initial and continuing training will be included to provide adaptable tools according to the pedagogical context.

• We expect some heterogeneity in the studies, which may limit our ability to conduct meta-analyses.

Introduction

Background

Most healthcare professionals, including those not working in psychiatry, must sometimes deal with psychiatric patients, at least for somatic problems. Opportunities for learning to do so are limited. Not all healthcare students have an internship in psychiatry, and even for those who do, it is necessarily limited to the mental illnesses encountered.^{1 2} Training in psychiatry requires specific skills and attitudes, which must be experienced and not simply memorised. Simulation training may be particularly adapted to this situation.

Medical simulation means ‘the use of a device, such as a mannequin, a task trainer, virtual reality, or a or a standardised patient to emulate a real device, patient, or patient care situation or environment to teach therapeutic and diagnostic procedures, processes, medical concepts, and decision making to a healthcare professional’.³ Simulation training is already widely used in several specialties and recognised as effective approaches to enhancing medical error management, patient safety and health professional team training.^{4 5}

Simulation in psychiatry mainly involves human simulation, defined as a ‘methodology that involves human role players interacting with learners in a wide range of experiential learning and assessment contexts’.⁶ Human simulation can be performed either with role play (RP) or with ‘a person trained to portray a patient in realistic and repeatable ways’.⁶ This person trained can either explore and facilitate a natural interaction with the scenario participant: it is named a ‘simulated patient’ (SP).⁷ Or this person trained has to provide a replicable scenario every time in case of examinations (eg, Objective Structured Clinical Examination (OSCE)): it is named a ‘standardised patient’ (StP).⁷ RP is based on the same principles, but the people who portray patient are not specially trained, and learners can play roles other than their own; in particular, professionals can play patients.¹ Other kinds of simulation have appeared recently in psychiatry, including virtual reality, voice hallucinations and use of manikins. Although RP has a strong history in psychiatry,¹ especially in the teaching of psychotherapy^{1 8–10} and in mental health nursing education,^{8 11} research on the effectiveness of simulation in psychiatry is less advanced than in other specialties.

Nonetheless, studies in psychiatric simulation have been appearing at an increasing rate. Some previous reviews have suggested the use of simulation in psychiatry for improving skills among students¹² and general practitioners¹³ as well as, more specifically, communication skills for those caring for patients with dementia.¹⁴ Nonetheless, the value of simulation in psychiatry remains controversial, especially for aspects such as the ability of SPs to accurately portray the cognitive, affective and behaviour complexity of mental disorders^{15 16}; relational authenticity^{15 16}; induction of symptoms in SPs.¹⁷ Most of these reviews have focused on particular simulation interventions without giving an overall picture of this field.1 15 18–22 For these reasons, we (a multidisciplinary team including psychiatrists, psychologists, epidemiologists and experts in simulation) decided to conduct a large systematic review of the effectiveness of simulation as initial and continuous training of healthcare professionals in the field of psychiatry. We will base our evaluation of effectiveness on the widely recognised Kirkpatrick’s Scale levels²³: learners’ satisfaction, change of attitude, skills, knowledge, behaviours, professional practices and benefit to patients. Our aim is to provide teachers, particularly of psychiatry, with exhaustive information and tools to implement or improve simulation programmes in this field, according to their individual and institutional needs and contexts.

Methods

This systematic review and meta-analysis will follow the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions ²⁴ and the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) guidelines.²⁵

Eligibility criteria

Types of study

To evaluate changes in attitudes, skills, knowledge, behaviours, professional practices and benefit for patients, we will include randomised controlled studies, non-randomised controlled studies and single-group pretest/post-test reports.

Only studies with only a post-test evaluation will be included for the assessment of participants’ satisfaction.

We will also include qualitative studies focusing on mechanisms, which may help to explore effectiveness in mental health simulation training.

We will exclude case reports, case series, cross-sectional and case–control studies, commentaries, editorials, opinions or purely descriptive articles, conference proceedings and review articles.

Participants

Inclusion

We will include studies that included all kinds of healthcare students and professionals receiving a simulation-based educational intervention in psychiatry:

• Those working directly in the field of psychiatry: psychiatrists, child and adolescent psychiatrists, geriatric psychiatrists, military psychiatrists, specialists in addictions, psychiatric nurses, psychologists, psychotherapists, nurses’ aides, social workers, speech therapists, art therapists and psychomotor therapists; and

• Those who may on occasion provide care to people with a psychiatric problem: psychosomatic physicians, all specialties of physicians, surgeons, nurses, aides and social workers (especially in paediatrics and general and emergency medicine, who must often care for patients with mental health problems), dentists and pharmacists.

Thus, all studies evaluating a simulation programme in psychiatry dedicated to healthcare professionals (including physicians, nurses, nurses’ aides, etc) will be included.

Interventions

We will include all forms of simulation-based education in psychiatry as detailed in table 1.

Table 1.

Different forms of simulation in psychiatry evaluated in this review^{1 35}

Technique	Definition	Example
Simulation using people as patients:
Standardised (StP) or simulated patient (SP)	These two terms ‘are often used interchangeably and refer to a person trained to portray a patient in realistic and repeatable ways. SP interact with learners in experiential education and assessment contexts’.6
→Simulated patient (SP)	It refers to a situation where the person has to explore and facilitate a natural interaction with the scenario participant.7	Attoe et al 201736: SP are actors with intellectual disabilities who play psychiatric disorders and provide on their experience.
→Standardised patient (StP)	It refers to a situation where the person has to provide a replicable scenario every time in case of examinations (eg, OSCE).	Hodges et al 199737: StPs are used to assess psychiatric clinical clerks.
SP educators (SPE)	SPEs are ‘those who work to develop expertise in methodology and are responsible for training and/or administering SP-based simulation’.6	Coyle et al 199838: SPEs act to teach psychotherapeutic skills to psychiatric residents and give them feedback.
Role play (RP)	Learners ‘asked to be someone quite different from themselves and, with little or no preparation, perform in front of peers and teachers’.1	King et al 201539: medical students played different roles (doctor or patient) in various psychiatric scenarios such as depression.
Simulation using virtual reality:	‘Computer-generated scenario or environment with which an individual can actively interact’.35 40
Virtual environment and patient	Psychiatric environment and avatar portraying a person living with a mental illness.	Lambert and Watkins 201341: used and proved the effectiveness of teaching nursing students appropriate communication methods.
Voice simulation	‘Use of sounds and voice through an electronic medium to portray the sounds encountered by a schizophrenic patient’.35	Wieland et al 201442: 74 students listened to audio recordings of common voices heard in schizophrenia while attempting to complete certain tasks such as a job application.
Simulation using manikins as patients (M)	Use of a high-fidelity patient simulator (HPS) to portray a person with mental illness.	Rabheru et al 201343: use of HPS to train residents in psychiatry in electric convulsion therapy.
Objective structured clinical examination (OSCE)	A rotation, followed by learners, through a series of stations with an encounter with an StP, model or other standardised task at each.1	Hodges et al 200244: description of several steps to implement different psychiatric scenarios in an OSCE.

We will include all these types of simulation, regardless of whether or not they are repeated or include feedback and regardless of the type of skill or process involved (diagnosis, clinical reasoning, psychopathological assessment and psychotherapeutic skills including but not limited to cognitive–behavioural therapy, family therapy, psychoanalysis, pharmaceutical prescribing, etc). We will include both simulation-based practice and assessment when used for educational purpose. That is to say, we consider that the evaluation step of a pedagogic programme with standardised patient (eg, OSCE) belongs to educational purpose, according to the learning effects of evaluation.²⁶ As a matter of fact, articles about assessment with StP as OSCE will be included. On the contrary, we will exclude all kinds of other non-educational use of simulations, considered as complementary to training and education. Thus, we will exclude studies that use simulation for non-educational purposes such as procedural planning, disease modelling or when SPs are used only to assess another pedagogical intervention without simulation or to assess some skills of healthcare professionals in real life.

We will include simulation training related to both of the following broad clinical topics:

• What is traditionally referred to as psychiatry: schizophrenia, psychotic and paranoid disorders, mood, anxiety, personality, eating and child development disorders, suicidal behaviour, attention deficit hyperactivity disorder and autism spectrum disorder.

• Psychiatric diseases that also involve other specialties: addictions, psychosomatic medicine, somatoform disorders and symptoms such as agitation, violence, delirium and so on.

Comparators

We will include trials comparing the interventions to:

• No training: students or professionals who are not exposed to simulation training;

• Other kinds of educational training with which simulation is compared.

We will also include trials with pretest/post-test measures and studies of adjuvant instruction in which simulation was added to other instructions common to all learners.

Outcomes

Primary outcomes

The primary outcome will be based on the Kirkpatrick criteria,²⁵ which WHO considers to be the standard reference for assessment of learning.²⁷

• Level 1: participants’ satisfaction;

• Level 2a: change of attitudes;

• Level 2b: change of knowledge, change of skills;

• Level 3: behavioural change;

• Level 4a: change in professional practices;

• Level 4b: benefits to patients.

For level 2b, we will differentiate changes of knowledge from changes of skills.

Secondary outcomes

To better identify the conditions needed to implement a quality simulation programme in psychiatry, we will also include secondary outcomes corresponding to specific features of psychiatry that have been mentioned in previous reviews^{1 15}:

• Quality of play-acting;

• Authenticity of situation;

• Authenticity of relationship;

• Specificity of communication in psychiatry;

• Psychiatric symptoms inferred during role play of a mental disorder and a potential care strategy;

• Balance between standardisation and improvisation in scenarios of mental disorders;

• Psychometric properties (validity, reliability and sensitivity) of tools used to assess simulation in psychiatry;

• Learning process for learners.

Patient and public involvement statement

Patient or public were not directly involved in the systematic review protocol. However, in the simulation area, patients have a wide place in training: to help the learners to experience a clinical situation within authentic condition; to take part in the feedback; until being ‘patient instructor’ (PI).

Thus, patient feedback on simulation (both on student performance and on their experience of simulation) belongs to the data that will be included in the systematic review. For the randomised controlled trials (RCTs) included, we will note when patients themselves assess the burden of the intervention.

We cannot disseminate results to all study participants included because we were not investigators. However, through publications in peer-reviewed journals, national and international conferences, we will try to make the results available for the maximum number of people.

Search strategy

Electronic search

A comprehensive search of the following electronic databases will be conducted: MEDLINE, EMBASE, Scopus, CINAHL, PsychINFO, ERIC, the Cochrane Library (Cochrane database of systematic reviews and Cochrane central register of controlled trials (CENTRAL)) and the Web of Science (Science and Social Sciences Citation Index). An epidemiology instructor experienced in systematic reviews (from Cochrane France: AD) and an experienced research librarian (AC) helped to design a Medline search query. Search terms included both MeSH terms and free-text words referring to simulation techniques, psychiatric practice, and mental disorders or symptoms. The search query for MEDLINE is reported in online appendix 1. It will be adapted to the other databases.

Additional search

We will search the tables of contents of the following journals for the last 10 years: Academic Psychiatry, Academic Medicine, International Journal of Medical Education, the Journal of Nursing Education, Nurse Education Today, Medical Education, BMJ STEL, Advances in Simulation, Simulation in Healthcare and Clinical Simulation in Nursing.

We will search for registers through the International Clinical Trials Registry Platform and websites (such as http://www.aspeducators.org (for the association of standardised patient educators), https://www.tripdatabase.com and http://www.greylit.org/).

Finally, we will screen all reference lists for further additional references.

No time limits will be set. Databases will be searched from inception. We conducted our search in December 2017. We will update our search if necessary.

No language will restrict the review.

Screening of identified studies

The main author (M-AP) will search all databases. Two review authors (M-AP, GG) will independently screen the titles and abstracts retrieved by search. We will obtain full reports for all references that appear to meet the inclusion criteria. The two review authors will then independently screen the full-text reports. All disagreements will be solved by discussion with the help of a third reviewer if needed to reach a consensus. We will use Covidence software and seek additional information from study authors where necessary to resolve questions about eligibility.

The review authors will not be blinded to the names of the journals, authors or institutions.

Data extraction

A standardised data extraction form, in Google form format, will be developed for collecting data from the selected studies. Data extraction will be carried out independently by two reviewers (M-AP, GG). In cases of disagreement, a third reviewer (CL) will help to reach a consensus.

We will extract the following characteristics:

• Study details: authors, year and journal of publication, year recruitment began, country;

• Study objectives;

• Participants’ characteristics: age, gender, inclusion and exclusion criteria, initial or continuing training, undergraduate or postgraduate (for initial training), profession (doctor, nurse, etc), healthcare specialty, number of learners;

• Methods: design and allocation, sampling, blinding, data collection time points, loss to follow-up, recruitment and retention rates, comparison/control group;

• Intervention:

  • Settings: location; simulated environment;
  • Description: type of simulation (RP, SP, SP educator, virtual reality (VR), voice simulation (VS), manikin (M)); recruitment for SP, SimP and PI (ie, real patients, volunteers or actor); method of allocation of roles for RP; simulation alone or with adjuvant pedagogy; pathology studied; process studied (clinical reasoning, therapeutic…); scenario summary; clinical variations; range of task difficulty;
  • Feedback: role of each person (eg, teacher, SimP, PI, observers), conduct;
  • Duration of each simulation, frequency, length;
  • Educational purpose (formative or summative evaluations or both; and for summative: OSCE or not), curriculum integration (links with global pedagogic programme).

• Outcomes: description, Kirkpatrick ranking, measurement instruments, evaluator (patient for SP, SimP and PI, or/and teacher and/or other learners, and/or student’s self-assessment), unit of measurement, timing of assessment and effects of intervention on the outcome.

In case of missing data, the corresponding authors of studies will be contacted for further information by email and two revivals without answer.

Risk of bias assessment

For quantitative studies, the risk of bias of RCTs will be assessed with the Cochrane Collaboration Tool designed specifically to assess this.²⁸ Selection, performance, detection, attrition, reporting and other biases will be assessed. Each of them will be scored as ‘high’ or ‘low’ or ‘unclear’ risk of bias.

For qualitative studies, the risk of bias will be assessed with the RATS scale.²⁹ Relevance of study question, Appropriateness of qualitative method, Transparency of procedures and Soundness in interpretative approach will be assessed. Each of them will be scored as ‘high’ or ‘low’ or ‘unclear’ risk of bias.

Two reviewers (M-AP, GG) will independently assess the risk of bias of included studies and any conflicts will be resolved through group discussion. A supervising reviewer (AD) will train the reviewers and help settle any disagreements to ensure the appraisal process is robust and transparent.

Data synthesis

For quantitative studies

The analysis of pre/post studies will be descriptive only. Meta-analyses will be done to pool data from RCTs.

Meta-analysis

We will compare the characteristics of the selected studies to determine the feasibility of performing meta-analyses. If meta-analysis is deemed possible, we will use both fixed-effects and random-effects models and compare their results. We will perform meta-analysis with RevMan V.5.3 software, generating ORs and 95% CIs for dichotomous data, and mean differences and SEs for continuous data. Forest plots will be produced.

Otherwise, when we encounter substantial content or methodological heterogeneity across studies, we will not conduct a meta-analysis but instead will use a narrative approach for synthesising the data. A systematic narrative synthesis will be provided with information presented in the text and tables to summarise and explain the characteristics and findings of the included studies. The narrative synthesis will explore the relations and findings both within and between the selected studies, in line with the ‘Guidance on the Conduct of Narrative Synthesis in Systematic Reviews’.³⁰

Evaluation of heterogeneity

We will assess statistical heterogeneity by visually inspecting the scatterplot of individual study effect estimates via forest plots and through the I-squared statistic.²⁴ This statistic reports the percentage of variability in effect estimates that can be attributed to heterogeneity rather than to chance. We will consider a value greater than 50% to show substantial heterogeneity.

Subgroup analyses

Subgroup analysis aiming at identifying potential moderators (ie, effect modifiers) will be conducted if possible, according to:

• Participants’ characteristics:

  • Initial or continuing training;
  • Undergraduate or postgraduate (for initial training);
  • Profession (doctor, nurse…);
  • Healthcare specialty (psychiatrists vs others).
• Intervention:

  • Type of simulation (RP, SP, SimP, PI, VR, VS, M);
  • Simulation alone or with adjuvant pedagogy;
  • Disorder studied;
  • Frequency and length of simulation;
  • Educational purpose (formative or summative or both; and for summative: OSCE or not);
  • Curriculum integration (of simulation thought to be in line with other teaching).

Assessment of reporting biases

If the meta-analysis includes 10 or more studies, we will draw a funnel plot and use the statistical test proposed by Egger et al ³¹ to investigate funnel plot asymmetry.

Sensitivity analyses

We will consider performing sensitivity analyses to explore the impact of risk of bias dimensions on the main outcomes of the review. Specifically, any sensitivity analysis will remove the studies judged to be at high risk of bias from the main analysis and evaluate whether results are robust.

For qualitative studies

A systematic narrative synthesis will be provided with information presented in the full text and tables to summarise and explain the characteristics and findings of the qualitative studies. The narrative synthesis will explore the relationships and findings both within and between the included studies, in line with the ‘Guidance on the Conduct of Narrative Synthesis in Systematic Reviews’.³⁰ Text mining of each abstract and thematic analysis of each full text will complete this analysis.

Quality of the evidence

The quality of evidence will be assessed with the Grading of Recommendations Assessment, Development and Evaluation,³² with gradepro software. The quality of the studies will be judged as high (further research is unlikely to change the confidence in the effect estimates), moderate (further research is likely to have an important impact on the confidence in the effect and may change the estimate), low (further research is very likely to have an important impact on the confidence in the effect and is likely to change the estimate) and very low (any estimate of the effect is very uncertain).

Reporting of the protocol and the review

This systematic review protocol will be reported according to the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P) guidelines³³ and the results according to the PRISMA Statement.²⁵ The results of the screening process will be provided in detail with the PRISMA flow chart. Quantitative data will be summarised in tables of individual studies, summary tables and forest plots, and qualitative data will be reported in tables and text.

Conclusion

This systematic review will provide relevant evidence about the effectiveness of simulation training in psychiatry. It will help to support instructors in implementing or improving these pedagogical tools that both form attitudes and teach knowledge and skills.

Our systematic review has the potential to have important effects in this field. Simulation may be particularly useful for training healthcare professionals in managing patients with mental health disorders in low-income countries or in places where very few psychiatrists are available. Moreover, it may enable the transformation of psychiatric training to adapt to changes in practices over the three past decades in many countries. Despite the change in emphasis from institutionalisation to ambulatory settings, psychiatric clerkships, internships and residencies have remained inpatient based.^{1 2 34}

Furthermore, exponential advances are taking place in virtual reality and artificial intelligence. This review will provide information about whether this progress is sufficient to meet the objectives of psychiatric simulation training.