A structured process to develop scenarios for use in evaluation of an evidence-based approach in clinical decision making

Abstract

Background and purpose

Scenarios are used as the basis from which to evaluate the use of the components of evidence-based practice in decision making, yet there are few examples of a standardized process of scenario writing. The aim of this paper is to describe a step-by-step scenario writing method used in the context of the authors’ curriculum research study.

Methods

Scenario writing teams included one physical therapy clinician and one academic staff member. There were four steps in the scenario development process: (1) identify prevalent condition and brainstorm interventions; (2) literature search; (3) develop scenario framework; and (4) write scenario.

Results

Scenarios focused only on interventions, not diagnostic or prognostic problems. The process led to two types of scenarios – ones that provided an intervention with strong research evidence and others where the intervention had weak evidence to support its use. The end product of the process was a scenario that incorporates aspects of evidence-based decision making and can be used as the basis for evaluation.

Conclusion

The use of scenarios has been very helpful to capture therapists’ reasoning processes. The scenario development process was applied in an education context as part of a final evaluation of graduating clinical physical therapy students.

Introduction

All 14 physical therapy programs in Canada now offer their clinical entry level programs at a Master level, with the last program transitioning from Bachelor to Master level education in 2012. The impetus for the degree change was primarily the realization that curricula were changing to include critical appraisal skills such as evaluation of the research literature in addition to clinical skills. The revised curricula train students to be consumers of research and to apply this knowledge in the clinical decisions they make for individual clients. The authors’ institution, like other physical therapy schools worldwide, adopted Sackett et al’s definition of evidence-based practice (EBP) which identifies three sources of information for clinical decision making: research evidence, client values, and clinical experience.¹ A model of practice and a clinical decision-making model were developed to guide the curriculum transition and both incorporate the three components of EBP.^2,3 The curriculum committees’ vision of the ideal Master of Science (Physical Therapy) (MScPT) graduate is one who practices using an evidence-based approach. To test whether the vision was realized, funding was received to conduct a research study to evaluate the EBP knowledge and skills of graduates from both the Master and the Bachelor programs.

A search of the literature exploring ways to evaluate use of an evidence-based approach in clinical decision making revealed a common thread: use of clinical scenarios as the basis for evaluation.^4–8 As such, in the context of this research study, it was also decided that clinical scenarios would be used, and a detailed process for the development of scenarios would be developed. The purpose of the scenarios is to provide a basis from which to evaluate use of the components of EBP in decision making. Some of the processes identified in the “key features” approach to scenario development were adopted into the current processes.^9–11 Key features scenarios are used in the assessment of clinical decision making in the Medical Council of Canada Qualifying Examination as well as the Royal Australian College of General Practitioners fellowship examination.¹² Key features are identified as “the essential elements in decision making” and in the case of scenarios developed to evaluate medical students’ decision making, they represent things like “recognizing dehydration.” Using the key features approach, scenario writing teams that included practitioners identified appropriate clinical domains and associated clinical problems to form the basis for the development of clinical scenarios and related questions.^9,11 The current scenario writing process similarly utilized the knowledge of practitioners/clinicians, but in this case key features were the different components of EBP in decision making. The aim of this manuscript was to describe the step-by-step process that was designed to develop clinical scenarios for use in the authors’ curriculum research study.

Scenario development

The groundwork

Embarking on the scenario development process, three conceptual decisions were made. First, it was decided that the scenarios would focus only on interventions and not cover prognostic or diagnostic aspects of physical therapy. This approach ensured that participants did not have to make a diagnosis before reflecting on the intervention choice. The focus on interventions simplified the process and provided clear boundaries to scenario writers by limiting the process to the intervention literature. Second, it was decided that the selected intervention in the scenario would be provided, rather than asking participants to decide on the intervention they would choose related to the scenario. This decision made the development of scoring rubrics easier, simplified the marking process for evaluators, and focused the participants’ preparation time so that they weren’t spending an inordinate amount of time searching for the best intervention. Finally, it was decided that two types of intervention scenarios would be developed – ones where the intervention had weak evidence and others where there was strong evidence for an intervention. This decision recognized a reality in physical therapy practice that not all interventions used in physical therapy have strong research evidence to support them. Using both strong and weak research evidence scenarios helped to determine whether participants use differential reasoning processes, depending on the strength of evidence for an intervention.

A four-step approach

Scenario writing teams made up of at least one clinician and one faculty member (or thesis graduate student who was a physical therapist) with expertise in the same clinical area used the process below to write scenarios. Clinicians were selected if they worked in one of the primary areas of practice (orthopedics, neurology, cardiorespiratory) and if they had at least 5 years of clinical experience. Table 1 illustrates the template developed to record the actions and findings of each step. Figure 1 summarizes the four steps in the scenario writing process.

Table 1.

Scenario name______________________
Scenario type
• Strong evidence – Level I or II evidence to support or refute intervention use
• Weak evidence – intervention supported only by Level III evidence or lower
	Description of requested information
1. Intervention	List intervention to go in scenario
2. Outcomes supported	List outcomes supported by strong research evidence. if able, also list common outcomes that may not be supported by strong research evidence
3. Evidence supporting intervention choices (3–10 evidence sources)	List references or evidence sources used to determine whether intervention supported by strong or weak evidence. include relevant clinical practice guidelines, if available
4. Most useful resources	List the three most important resources
5. Scenario framework information	If available, provide participant characteristics, parameters of intervention (eg, intensity information, number of repetitions, times per week), and treatment setting information
6. Search terms	List up to 15 appropriate search terms specific to the scenario
7. Databases	List up to 10 appropriate databases
8. Client’s primary concern	Describe the client’s primary concern so that participants can pull out the client aspect of evidence-based practice when considering the scenario
Other comments

Figure 1.

Scenario development process.

Step one: identify prevalent condition and brainstorm possible interventions

Prior to scenario writing, clinicians involved in the scenario writing process reflected on their practice and the conditions that they treated most frequently. This information was the primary information used to determine conditions on which the scenarios would be based. Specific epidemiological data regarding conditions physical therapists treated were not available for Alberta and were not used to guide this process. However, the yearly provincial report describing areas of practice of physical therapists was considered and conditions from areas in which the majority of physical therapists worked were generally chosen (ie, general practice, orthopedics, and neurology where at least 72% of therapists worked, not women’s health where only 0.3% of physical therapists worked). Development of scenarios for obscure conditions was avoided, even if excellent evidence was available to support an intervention. After conditions were identified, each team brainstormed possible interventions. Interventions were linked to well-established physical therapy goals. For example, one of the goals for someone with stroke might be to increase walking speed, thus interventions to consider for the scenario included strengthening exercises, treadmill walking (with and without body weight support), overground training, and circuit training. Interventions chosen were ones that were generally available in most clinical settings. For the example, treadmill interventions were considered as an intervention but Lokomat® (Hocoma, Volketswil, Switzerland) was not, as very few places have the Lokomat available.

Step two: literature search

After one to three possible interventions were identified, the scenario writing team searched the literature to find research literature pertaining to the interventions. Searches included the research literature, as well appropriate clinical practice guidelines. If a systematic review was available, it was used as a starting point for subsequent searches (ie, it helped guide literature retrieval at the level of the individual study). Search terms were selected based primarily on characteristics of the intervention, and broadly on the participant characteristics. The search terms and databases used were recorded on the template (Table 1). The search strategy used in the scenario development process is different from the traditional use of the patient–intervention–comparison–outcome (PICO) search format. The PICO search format starts upfront with a focused clinical question that includes specific information about the participant, the intervention, a comparator intervention, and possible outcomes in order to identify pertinent literature.¹³ Instead, we used clinical experts’ knowledge to identify potential interventions and then searched for the most pertinent literature to support or refute the use of the intervention. Thus, the search strategy was a truncated PICO search and included only information about the patient/condition and the intervention. During the scenario development process, clinicians indicated that this abbreviated process more closely mirrored what they actually do in clinical practice if they do a literature search.

The result of the literature search process was determination of the level of evidence (weak or strong, as defined below) related to the identified interventions. The studies and clinical guidelines identified in the literature search related to the interventions were evaluated and assigned a level of evidence using the classification framework developed by the American Academy for Cerebral Palsy and Developmental Medicine.¹⁴ Interventions with Level I and II evidence to either support or refute their use (ie, systematic review, large randomized controlled trial, or smaller randomized controlled trials with wide confidence intervals) were used to develop strong evidence scenarios. Interventions determined to have Level III evidence or lower, representing more inconclusive evidence, were used for the development of weak evidence scenarios. Of the literature and clinical guidelines identified, scenario writers were asked to provide their three most important resources with respect to the intervention (ie, those resources that particularly helped to determine level of evidence, or provided information for the scenario framework).

Step three: develop scenario framework using information from the literature

After the level of evidence for the interventions was determined and one to three seminal articles were identified, the scenario writing team worked from this literature base to extract participant characteristics, treatment outcomes, and treatment settings that could be used to develop the scenario framework. All of these components are included in the PICO system for literature searches;¹³ the method the authors’ students are taught during their program. Thus, by systematically pulling out information about these components from the literature, the probability that participants would find the appropriate literature during their literature searches was increased. The scenario writing teams were diligent about checking the evidence with respect to all aspects of the scenario framework, including outcomes, participant characteristics, and intervention contexts or characteristics. Some interventions had strong evidence for all outcomes discussed in an article and others had strong evidence for only one or two of many outcomes. For example, cardiac rehabilitation reduces mortality and hospital admissions, but not the incidence of myocardial infarction.¹⁵ Therefore, the scenario framework for the cardiac rehabilitation strong intervention scenario used reduction of hospital admissions as one of the patient goals. Constraint-induced movement therapy is a good example of an intervention where many different aspects of the scenario framework (eg, stage of recovery from stroke) had an impact on strength of the evidence for the intervention. It also provides an example of the need for frequent review of the literature. For example, in a Cochrane review, evidence was strong for a reduction in upper extremity disability with constraint-induced movement therapy, ¹⁶ but a more recent publication by the same author reports no effect on disability.¹⁷

Step four: write scenario

The final step was composing the scenario. The scenario writing teams were instructed to keep the scenario short (ie, 200 words or less) and to use lay language. Client values, one of the three pieces of EBP,¹ were embedded in the scenario to be used for goal setting questions. The problem of the client described in the scenario had to represent the component of activity rather than impairment, as defined by the International Classification of Function, Disability, and Health.¹⁸ The results of standardized outcome measures to describe the clients’ abilities were discouraged; instead, scenario developers were encouraged to describe the parameter in a functional context. For example, instead of presenting results for distance walked in a 6-minute walk, the scenario described how long it took the client to walk to the corner store three blocks away.

An example scenario

One of the scenarios developed was for falls in the elderly. Falls was chosen as a condition because it is prevalent (up to one-third of older adults over the age of 65 years, will fall each year)¹⁹ and because it is a condition that crosses different practice settings (inpatient, outpatient, and home health as well as neurology and orthopedic). The literature search led to a Cochrane review and a clinical practice guideline that both provided strong evidence (as graded by the American Academy for Cerebral Palsy and Developmental Medicine system) for multicomponent exercise interventions that include strength and balance activities.^19,20 The scenario reflected client characteristics extracted from the literature. Client goals were described that matched outcomes supported in the literature. The intervention was provided, but details of the intervention were not (eg, amount of balance challenge required to be effective). Details of the intervention were a possible discussion topic during the participant interview.

You are a consultant at a senior living complex. Tanya is a 68-year-old resident there. She lives independently, doing most tasks herself except for outdoor ones such as cutting the grass and snow shovelling. Tanya walks independently around the building with no walking aids and she drives her car short distances. Tanya has had a couple of falls in the last 2 months in her apartment. The home care liaison at the seniors’ complex is concerned about Tanya’s falls – they are the first falls she has had. Tanya and the home care liaison want to prevent future falls and the home care liaison asks you to see Tanya. You decide to set her up with an exercise program that includes strength and balance exercises.

The participants’ task, after reading the scenario, was to decide on the effectiveness of the provided intervention. During the interview discussion the processes and resources they used to make the decision about intervention effectiveness were queried.

Discussion

Clinical scenarios are ubiquitous in both physical therapy research studies and in education programs. They have been used in research as a basis to evaluate specifically the use of evidence in decision making,⁴ and this was the purpose of the scenario development in the current research. Scenarios have also been used more generally to evaluate EBP knowledge and skills.⁵ Scenarios used in education contexts help students synthesize physical therapy theory and knowledge in relation to a clinical example. The frequent use of scenarios in physical therapy and health care education in general called for a standardized process of scenario writing. A logical and effective process to develop scenarios that can be used in research and education was developed and described. The well-defined steps in the scenario development process have allowed various people including clinicians, thesis graduate students, and faculty members to contribute to scenario writing for the research study. The scenario development process is applicable to any area of physical therapy, but the process can be used – without adaptation – in other areas of health care such as occupational therapy, nursing, or medicine where scenarios may also be useful as part of evaluation of clinical decision making.

The process used findings from the literature search step to extract elements for the scenario framework (ie, participant characteristics, outcomes, intervention characteristics). Embedding these scenario framework elements extracted from the most appropriate literature within the scenario increases the likelihood participants will identify appropriate literature, probably similar to the literature the scenario writing team identified. Directing therapists to appropriate literature helps to increase efficiency of evaluation (ie, evaluators may need to do less literature checks) and it may be easier to ensure scoring is equitable. Providing the intervention in the scenario also made evaluation easier, as the evaluator was able to focus the evaluation on the responses to the one provided intervention, instead of the multitude of interventions that participants may have selected related to the scenario.

Because the scenarios are linked to the literature, one of the biggest challenges is keeping them current. For example, since the first iteration of scenarios was written, a Cochrane systematic review of Down’s syndrome and treadmill training (one of the authors’ scenarios) was published.²¹ This new publication did not change the level of evidence from treadmill training (it was still strong), but the article was added to the scenario template as evidence supporting the intervention. The project coordinator has been able to do most of the scenario updates using the well-established process.

It was also challenging to determine the quality of evidence because authors of systematic reviews or clinical practice guidelines often used grading systems that were slightly different. Two people conferred to agree on the strength of the evidence using the American Academy for Cerebral Palsy and Developmental Medicine guidelines for coding levels of evidence. If those two people were not able to agree, one of the researchers worked with the team to come to a decision. This was an important step as scenarios had interventions with either strong or weak evidence, and it was crucial to make the correct interpretation. A conscious effort was made not to describe outcomes as change scores on standardized measures. This is different than information provided in scenarios to students when they are early in their training, when information on standardized measures is intentionally provided so they can practice interpretation. The expectation of physical therapy graduates (the population interviewed) is that they will think beyond change scores on outcome measures to functional outcomes that matter most to the patient, thus more functional information than what was provided in the scenarios. Scenarios related to interventions with weak evidence were more challenging to craft than scenarios with strong evidence for an intervention. For scenarios with strong evidence for an intervention, there was often a Cochrane review or some other well done review upon which to gather information about patient characteristics and outcomes with strong evidence. For weak evidence scenarios, it was a longer process to find and particularly to interpret the findings from the literature. Weak evidence scenarios may be more challenging for participants as well. Depending on whether the intervention in the scenario was supported by weak or strong evidence, it was hypothesized that therapists may utilize the components differently (ie, report the need to use clinical experience more in the event that research evidence is poor). Scores will be compared for weak and strong evidence scenarios.

Application of the process to an education context

The scenario development process used for the research study has recently been utilized in the authors’ MScPT education program as part of the final evaluative exercise for clinical physical therapy students. Like many programs, previously the students worked in groups to complete a small research project, or a systematic review in a specific area. These projects were demanding for both faculty and students and did not match well with the goal that graduates of the program be excellent consumers of research knowledge. Based partly on the experiences with the current research study of program graduates, the final evaluative project for MScPT students has been changed to a 2-week group project using clinical scenarios to evaluate EBP knowledge and skills. The scenario development process used in the context of research has now been adapted and used in this education context. Students in small groups receive a clinical scenario with questions. Some of the questions for both the research study and the student final evaluative exercise have been adapted from the Fresno test.⁵ Students provide written answers to the questions within a 2-week timeframe and then each individual student participates in a 10-minute oral exam to evaluate their individual knowledge. Both strong and weak evidence scenarios have been retained in the MScPT final evaluative exercise. A key component of an evidence-based approach is knowing what to do when strong research evidence is not available to guide treatment. Especially early in the clinical training, students often have great difficulty formulating a treatment plan when there is no evidence, and their clinical experience is minimal. By providing weak evidence scenarios, students’ overall approach and progress towards a balanced approach to clinical decision making is intentionally being evaluated. Differences in student responses and assignment scores between weak evidence and strong evidence scenarios will be explored.

Conclusion

The well-defined scenario development process used in the curriculum research study has helped to continuously evaluate scenarios and write new ones as the study progressed. There is now a bank of more than 20 scenarios from various areas of clinical practice, all structured and developed similarly so that they fit within the evaluation process. The scenarios are a strong basis from which to discuss therapists’ clinical decision making and the components of EBP they may or may not use in their decision making. Future work using scenarios will increase an understanding of the degree to which EBP skills and knowledge are transferred from an education to a clinical context.