Skip to main content
Applied Clinical Informatics logoLink to Applied Clinical Informatics
. 2016 Mar 30;7(1):191–210. doi: 10.4338/ACI-2015-08-RA-0111

A Design Methodology for Medical Processes

Simona Ferrante 1,, Stefano Bonacina 2, Giuseppe Pozzi 1, Francesco Pinciroli 1,3, Sara Marceglia 4,5
PMCID: PMC4817343  PMID: 27081415

Summary

Background

Healthcare processes, especially those belonging to the clinical domain, are acknowledged as complex and characterized by the dynamic nature of the diagnosis, the variability of the decisions made by experts driven by their experiences, the local constraints, the patient’s needs, the uncertainty of the patient’s response, and the indeterminacy of patient’s compliance to treatment. Also, the multiple actors involved in patient’s care need clear and transparent communication to ensure care coordination.

Objectives

In this paper, we propose a methodology to model healthcare processes in order to break out complexity and provide transparency.

Methods

The model is grounded on a set of requirements that make the healthcare domain unique with respect to other knowledge domains. The modeling methodology is based on three main phases: the study of the environmental context, the conceptual modeling, and the logical modeling.

Results

The proposed methodology was validated by applying it to the case study of the rehabilitation process of stroke patients in the specific setting of a specialized rehabilitation center. The resulting model was used to define the specifications of a software artifact for the digital administration and collection of assessment tests that was also implemented.

Conclusions

Despite being only an example, our case study showed the ability of process modeling to answer the actual needs in healthcare practices. Independently from the medical domain in which the modeling effort is done, the proposed methodology is useful to create high-quality models, and to detect and take into account relevant and tricky situations that can occur during process execution.

Keywords: Healthcare process modeling and design, computing methodologies, patient-centered care, patient care management, critical pathways

1. Introduction

Healthcare processes are acknowledged as complex [1, 2] and need transparency [3] of all the process elements (structures, actors, i.e., all of the process participants [4], roles, activities, and constraints) to achieve their implementation.

Healthcare process modeling is considered a possible solution to break out complexity and provide transparency. Indeed, healthcare process modeling is regarded as the basis for:

  1. establishing shared protocols for patient’s care;

  2. facilitating the compliance to shared protocols, thus limiting problems due to incomplete communication or misunderstandings among different actors;

  3. monitoring deviances, redundancies, and failures of the protocols, thus early identifying problems that could lead to un-prevented errors; and

  4. understanding the information flow, thus identifying requirements and specifications for information system re-engineering and interoperability [5].

For example, the lack of interaction and understanding among humans or among systems is a major problem in the context of preventable medical errors producing damages and high economic costs in many hospitals [6–8]: partial or untimely information exchange, and misunderstanding among heterogeneous actors can cause delays and difficulties in following shared protocols. Process modeling is able to track and describe cooperative work [8], thus helping to overcome the information gap among different actors.

The importance of process modeling in healthcare is also implied by a recent statement of the American Medical Association (AMA) [9], pointing out the eight priorities to improve electronic health records (EHRs) usability. The AMA believes that supporting team-based care through performing work as necessary and to the extent their licensure and privilege permit as well as by dynamically allocating and delegating works is one of these priorities [9]. Process modeling, by representing and clarifying the actors, their roles, and their actions and resources, have a primary role in achieving this priority. Similarly, promoting care coordination by automatically tracking referrals and consultations and offering product modularity and configurability are listed as great priorities to let EHRs follow the patient’s progresses [9]. Process modeling, by tracking the information flow throughout the process, and by identifying its core phases, is a fundamental support in the design of Information Technology (IT) systems supporting care coordination and providing modular solutions. The modeling approach can be also beneficial in ensuring the integration and interoperability among heterogeneous information systems [10–12]: the application of standard-based solutions is facilitated by a reliable description of the process underlying the information exchange.

All together, these considerations highlight the need of robust methodologies to design and implement process modeling in healthcare. The aim of the paper was to provide a design methodology specific to healthcare processes and apply it to an exemplar stroke rehabilitation.

2. Requirements characterizing the medical domain

The healthcare domain is recognized as highly complex and extremely variable [2]. The characteristics of the healthcare domain make it unique, and need to be carefully considered when modeling healthcare processes [2]. Clinical knowledge, clinical information, clinical data, variability, time, patient-centeredness, multiple stakeholders, and patient safety are altogether recognized as the hallmark of healthcare domain [2]. When modeling healthcare processes, the above characteristics become requirements for the modeling approach [13–15]. Moreover, both clinical processes and organizational processes introduce a dynamic nature of activities strongly requiring adequate modeling methodologies [8, 14, 16].

Organizational processes mainly deal with business aspects in healthcare organizations. Clinical processes instead deal with clinical decision-making and involve the professional judgment and experience, the uncertainty due to patient’s response, and the compliance to treatment. Moreover, healthcare processes, as for example medication prescription, often require that the clinical and organizational domains interact and cooperate within the same macro-process [17].

Table 1 and ‣ Table 2 summarize the main complexity areas of both clinical and organizational processes. For each of them sources of complexity (i.e., sub-areas) are specified that become requirements for the modeling effort. A pre-condition to select the proper clinical pathway is the patient’s diagnosis.

Table 1.

Modeling requirements to represent the characteristic features of the clinical domain; R= requirements.

Clinical process complexity areas Sub-areas Modeling requirement to represent the sub-area Ref.
Medical knowledge Evidence based medicine, guidelines, recommendations R1: Compliance to guidelines [8,14,15,20,34-40]
R2: Deal with dynamic evolution of medical evidence-based knowledge [8,41]
Local practices R3: Manage flexibility (from meta-model to local context) [44]
Clinician personal experience, habits and skills R4: Bind the design to interaction and cooperation between analysts and domain experts [45]
Learning by practice R5: Deal with learning curves (process change) [44, 45]
Building evidence from practice R6: Foresee process mining with big data analysis [25, 26]
Response to treatment Timeframe short-term or long-term response R8: Manage uncertainty [46–48]
Compliance – depending on patient’s engagement R9: Manage indeterminacy [46–48]
Expected outcomes R10: Define outcome variables [49, 50]
Patient feedback to reshape therapy – depending on patient’s empowerment and education R11: Integrate patient-reported outcome measures [49, 50]
R8: Manage uncertainty [46–48]
Personalization of care Patient-centric approach – treatment definition considering patients preferences and history R7: Manage exceptions [42]
R3: Manage flexibility [34]
Treatment adaptation considering occurring changes R3: Manage flexibility [34]

Table 2.

Modeling requirements to represent the characteristic features of the organizational domain; R=requirements.

Organizational process complexity areas Sub-area Modeling requirement to represent the sub-area Ref.
Systems and infrastructures Compliance with standards R12: Manage adaptivity and dynamic evolution (the model should include changes without losing information) [37, 42, 51, 52]
R13: Implement interoperability [10]
Introduction of local IT infrastructural constraints R3: Manage flexibility (from meta-model to local context) [41, 43]
Comparisons between different organizational systems (also to model system portability) R14: Introduce metrics [54]
Actors Related actions, multiple profiles executing the same task R15: Assign responsibilities [53-56]
Data access R16: Define privacy and security constraints [10, 23]
Domain knowledge of the specific actor R17: Integrate dictionaries and standard terminologies [57]

Medical decision-making, also known as clinical problem solving [18], is the basis of clinical processes, acting in all the phases of care (prevention, diagnosis, treatment, and rehabilitation). Decision-making usually relies on the healthcare profession’s medical knowledge, even though the notion of shared decision making (including patient’s participation) has emerged in the last years [19]. In addition, medical decision-making depends on the patient’s response to treatment and on the need to adapt the intervention based on patient’s characteristics (personalization of care). Medical knowledge is a general term including evidence-based medicine, guidelines, and international recommendations, as well as the clinician’s personal experience [20] that builds on personal skills and learning curves, local practices within the specific clinical environment, and on the evidence that is built upon the everyday practice. Apart from the need to include guidelines and recommendations [21], the evolving nature of medical evidence-based knowledge requires that models should deal with dynamic evolution. The notion of evolution has been defined by Reichert mainly in terms of organizational processes and infrastructural standards as the ability of a process to change when the business process evolves [22]. The concept should be enriched by the notion of dynamic, meaning that the model has to face process variants and deviations, too. Such variants and deviations are due to relevant discoveries in the clinical context, as well as to the personal evaluations from clinicians who may customize the pathways suggested by guidelines and recommendations [21–23]. Therefore, optional activities, corrective measures and exceptions are more frequent and less predictable in clinical pathways than in non-medical processes [24]. The personal experience that drives medical decisions introduces also the big challenge of the translation of such experience into the process model.

Graphical modeling languages provide a shared and understandable way to represent processes and can be used to facilitate context analysis and to translate experiences into models [19, 20, 22]. Moreover, process mining techniques – a methodology to identify and reconstruct process models starting from event logs – can be successfully applied to discover hidden clinical pathways based on intrinsic knowledge, and to translate them in sharable knowledge [25, 26].

In healthcare, local practices may differ from clinical center to clinical center, implying that the general meta-model, valid for the whole domain, should be instantiated including local constraints (from habits to available resources/infrastructures). This is arranged through flexibility [27, 28]. Flexibility also helps when the modeling effort is devoted to process optimization and requires changes in everyday practices. In fact, a flexible model can decrease the resistance to changes as well as the learning curve of the new process settings introducing smoother implementation steps [29].

Major complexity issues in the evaluation of the patient’s response to treatment are the expected response time, the patient’s compliance, the expected outcomes, and the feedbacks reported by patients on the treatment/therapy. So, patient’s response to treatment is a critical decision-making influencing factor that introduces uncertainty. Whereas patient’s compliance depends mainly on patient’s engagement and education, response time depends on patient’s clinical condition or on the therapy per se, thus introducing possible undetermined waits in the model execution until the response is reached (indeterminacy). The definition of outcome measures is crucial since the model should be able to monitor possible risky situations: outcome measures can be hence used as verification points producing alarms/alerts. The current patient centric approach to care considers the patient as an active actor in every medical process. The patient is hence often demanded to provide a feedback on therapy (patient-reported outcome measures, [30]). This introduces another type of uncertainty due to the level of patient literacy and empowerment that can affect the overall reliability of the patient-reported feedback [31].

Finally, personalization of care refers to the need to adapt the general evidence-based or experience-based behaviors to the patient’s specific conditions. This could bring to both the need of adaptation (already defined as flexibility) and to the generation of exceptions that the model should be able to manage [8].

Organizational processes deal with business management, at the system or infrastructural levels and at the actor level. Healthcare processes have however some specific features also within the organizational domain that become requirements for the modeling effort (‣ Table 2). At the infrastructural level, since one of the main issues in healthcare IT is the heterogeneity of available systems [10, 23], interoperability and adaptation have to be introduced into the model through the compliance with technological and health-oriented standards [10, 30].

From the organizational viewpoint, the need to deal with numerous actors becomes increasingly important [2, 8]. The model should, therefore, include the management of responsibilities (an action can be performed by many actors, but only one is responsible for it), introduce privacy and security constraints to ensure the protection of healthcare data, and integrate dictionaries and standard terminologies to deal with heterogeneous actors [32, 33].

Hence, modeling healthcare processes is more complex than modeling non-medical processes. Medical processes are characterized by uncertainties, unpredictability, evolution, higher variability and difficult generalization [19, 24, 32, 33] that the design methodology needs to address.

3. The Design Methodology for Healthcare Processes Modeling

Provided the complexity of the domain, healthcare process modeling needs to start with an in-depth analysis of the environmental context (phase 1, ‣ Figure 1). This step requires the involvement of healthcare and organizational/administrative professionals within the hospital organization, as wrong domain assumptions [58] or management errors [59] can produce unwanted consequences. After that, the process designer, that is the responsible for process modeling, proceeds with the definition of the conceptual model (phase 2, ‣ Figure 1) of the process and its mapping to a logical model (phase 3, ‣ Figure 1) [21, 22]. Every phase may include, in turn, some internal sub-steps.

Fig. 1.

Fig. 1

The design methodology includes 3 main phases, each including sub-steps.

For a good design, the modeled process should be predictable, repeatable, distributed, automatable, and feasible. Predictable means that the behavior of the process is clearly defined a priori and structured. Repeatable means that, under the same conditions, every process instance will evolve according to a shared process model, and will experience the same behavior. Distributed means that the process can be distributed among different execution and organization units. Automatable means that the process can be supported by automation for what concerns timing, schedules and task execution. Finally, feasible means that the process involves applications that can be easily implemented [23].

3.1 The Context analysis

The analysis of the environmental context (phase 1) includes the identification of the available sources of information (such as recommendations, clinical guidelines, local practices) for a full understanding of the domain of interest (Domain Analysis), the selection of the formal notation, and the definition of all the most relevant and critical aspects, especially as reported by healthcare professionals, that the model should be able to manage (Perceiving Significant Issues, in ‣ Figure 1).

Healthcare and organizational/administrative professionals within the hospital are the domain experts, knowing how processes are enacted in practice. These professionals are however unfamiliar with formal notations. Conversely, model designers are experts of the process representation by formal notations. The use of a graphical modeling language to design and visualize the process model has a twofold goal: designers produce a high-level and abstract schema, which does not consider the implementation issues; and process participants, that are not expert in business analysis, are facilitated in understanding the process representation and in providing useful feedbacks and evaluations. Therefore, graphical modeling languages may provide a powerful mean to improve the healthcare professionals/analysts communication and interaction.

The Business Process Model and Notation (BPMN) language is useful to describe the sequence of activities, that is the normal flow of execution, as well as the resources and the actors, that are the executing units which can be involved in the execution of the process [60, 61]. Each actor is reported in a separate swimlane and all the tasks within the same swimlane require the same executing actor. BPMN graphs are often used to describe the care pathways [60, 61], but the use of these diagrams alone [62] limits the potential of process modeling on improving healthcare delivery [63].

However, to cover more facets of the process model than those captured by a BPMN graph, other diagrams and representations are needed. Indeed, the Unified Modeling Language (UML) graphical notation offers activity diagrams that are the equivalent of BPNM graphs, but also provides other views on the process that can be used to detail different dynamic and static aspects. BPMN and UML 2.0 activity diagrams have been recently compared by three different evaluation criteria [64]: the ability to be immediately understandable to the reader, the adequacy of the graphical syntax to represent business processes, and the ease of mapping to an executable language for business processes. For the first two evaluation criteria, BPMN and UML 2.0 activity diagrams were considered comparable. For the third issue, BPMN diagrams can be automatically mapped to a Business Process Execution Language, while UML 2.0 activity diagrams could not. This disadvantage of UML 2.0 activity diagrams did not prevent the use of UML in our study. Additionally, UML comes with more diagrams providing the reader with a much rich description of the entire application domain [47, 65–67]. Among UML diagrams, we mention here:

  • the use case diagrams which describe the behavior of a system by the user view allowing to educe high level system requirements;

  • the class diagrams which describe the structure of the information model managed by the process in terms of classes (i.e., entities of the real world represented by their characteristics), as well as that of the external databases references by the process;

  • the sequence diagrams which describe the dynamic interactions among objects (i.e., instances of classes) highlighting the temporal flow of messages exchanged. These diagrams can be used to describe specific scenario including cyclic behaviors or exceptions.

As a consequence, we opted for UML.

3.2 The Conceptual Modeling

The conceptual modeling phase (phase 2) is composed by pre-modeling, modeling, and expert validation. Pre-modeling identifies all the goals to be achieved, as well as the interactions with any external information system (EIS) and the technological infrastructures. The Modeling sub-step describes the functional aspects, the organizational aspects, and the business aspects formally. First, functional aspects include the activities, objects, and data managed by the healthcare process; abnormal situations that may occur during the execution of a process; transactions and compensations to be adopted would an activity fail. The organizational aspects describe the actors, roles, skills, authorizations over activities and managed information. Finally, the business aspects describe the goals of the healthcare process. Generally, activity centric modeling exploiting a top-down approach is the most used and efficient modeling strategy [23]. This strategy implies that modeling starts from a high level view of the process, and then refines all the details of every module and component, increasing the specification towards the identification of basic activities.

The expert validation is needed to strongly check that the designed healthcare process corresponds to the needs and to the practices the domain experts (i.e., healthcare and organizational/administrative professionals) are familiar with. The validation of the model includes two main phases. First, the analyst has to assess if the designed model is syntactically correct in order to validate it as a starting point for the logical modeling towards the software implementation. Then, the experts of the domain have to check if the model is semantically correct by means of an iterative approach based on interviews and/or focus groups. The analyst has to meet groups of domain experts (process participants), grouped by type (i.e., only doctors, only therapists and then nurses…), to validate the flow of information in the simplest activities of the process; then a final focus group including one representative per each type of process participant is required to test the whole model. An iterative refinement of the model is performed until no further request comes from the domain experts. Only when the iterative refinement approach is finished and a total agreement between experts and analysts is reached, the model can be considered as final and used as specifications for the software system.

3.3 The Logical Modeling

The logical modeling (phase 3) aims at translating all the conceptual issues resulting from the conceptual model to the execution language of the process execution environment. In fact, every execution environment comes with its own executable language, and the conceptual model must be translated accordingly. In this phase, the process designers and implementers interact with the health IT professionals working within or for the hospital organization. The final product, i.e., the process represented by the execution language, may become a module of the Hospital Information System (HIS).

4. Case Study

To translate into practice the methodological approach previously described, we chose the process of stroke rehabilitation, as an exemplary case. Specifically, within stroke rehabilitation, we tested the sub-process of patient progress assessment/evaluation and its integration within the HIS. The case study referred to a real environmental setting, the Physical Medicine and Rehabilitation Unit, Fondazione Salvatore Maugeri, Research Institute of Lissone, Italy. It offered specialized rehabilitation to about 72 patients a day, having 54 inpatient beds. Every year about 70 stroke patients were treated. The staff included 6 medical doctors specialized in Physical Medicine and Rehabilitation, 1 psychologist, 13 physiotherapists, 1 occupational therapist, and 18 nurses. The modeling effort aimed to:

  • establish a shared protocol for the patient’s progression assessment/evaluation;

  • facilitate the compliance to a shared protocol, limiting problems due to incomplete information sharing/exchanging or misunderstandings among different healthcare professionals (increased transparency);

  • monitor deviances from the protocol, redundancies, and failures, thus early identifying problems on the rehabilitation process.

This case study presented all the typical complexity areas identified in Section 2 concerning the clinical processes, as detailed in ‣ Table 3.

Table 3.

Description of the case study process in the light of the complexity areas characterizing clinical processes.

Clinical process complexity area Sub-area Case study instantiation Effect on the model
Medical knowledge Evidence based medicine, guidelines, recommendations National Guidelines Clearinghouse [68], where clinical guidelines from Australia [69], United States of America[70], Scotland [71], and Canada [72] were the main evidence-based recommendations used. The selected guidelines were the base of the pre-modeling meta-model and were included as general constraints to be followed in the protocol definition
Earlier versions of the guidelines were also considered in the model, since patients may have started the rehabilitation process before the last version of the guidelines/recommendation was issued. Each guideline was labeled with the validity date, and the specific task in the evaluation process was connected to the date.
Local practices Textual and graphical clinical pathways were developed locally and used as main reference. The local pathways were remapped on the guidelines.
Clinician personal experience, habits and skills The personal habits, expertise, and information needs of the different actors (clinician, nurse, and therapist) were considered in the model. All the process participants were interviewed during context analysis to define all the information processed so that it could be included in the model.
Learning by practice There are established and sometimes sub-optimal behaviors (e.g., clinical scales recorded in paper-based patient’s diaries) that the modeling methodology can highlight and that can be improved by implementing a new software module. The model should represent the actual practices without introducing changes that imply specific training. It should be validated with the process participants before the release.
Building evidence from practice To evaluate the compliance to guidelines, the model included structured and organized information, that can be collected for secondary use. The model included the collection of logs.
Response to treatment Time frame – short-term or long-term response Rehabilitation is usually characterized by long-term response with an alternate outcome. The model allowed a variable waiting time between two actions (e.g., introducing a control on outcomes and not on time) and the possibility to map different responses (e.g., using forks) to map different behaviors.
Compliance – depending on patient’s engagement All the rehabilitation exercises conducted not in constant presence of the therapist can be ineffective due to patient’s misunderstanding of the indications or exercise descriptions. This problem becomes crucial when the therapist has to teach hoe therapy to the patient. The model considered that some activities need to be repeated and included the management of educational information for the patient at home (i.e., booklets).
Expected outcomes Outcomes are essentially clinical scales evaluating functions during activities of daily life, independence quality of life and mental state. Both the outcomes measures reported in the recommendations and the ones used locally were considered in the model. The model included the organization and the electronic collection of the outcome variables. A data structure for the outcome measures used locally was defined.
Patient feedback to reshape therapy – depending on patient’s empowerment and education Psychometric questionnaires to report directly the patients’ feedback about how they feel or about a specific function they need to rehabilitate were considered in the model. Patient-reported outcome measures were integrated in the software to collect clinical scales.
Likert scales scoring directly how the patients perceive their status before and after therapy (i.e., the general perceived effect, GPE) were included in the model. GPE was integrated in the software to collect clinical scales.
Personalization of care Patient-centric approach – treatment definition considering patients preferences and history There are some patient-dependent cautions to be considered during the rehabilitation of specific functions, e.g., required aids or level of assistance… that should be considered in the model. The possibility to carry out the same treatment using different aids or different level of assistance was modeled in the treatment class.
There are some patient dependent cautions to be considered in the definition of treatment duration and schedule e.g., patient ability to be focused to the training, patient resistance and strength. A dynamic setting of treatment duration and schedule was included.
Treatment adaptation considering occurring changes Consider possible interactions between the patient comorbidities and the chosen treatment, that may become evident after clinical pathway definition. The possibility to manage discovered interactions as alternative pathways in current treatment was included.

4.1 Phase 1: Environmental Context

The first sub-step was the domain analysis. The main international guidelines on cardiovascular patients who suffered from a stroke were identified. The main source of evidence was the National Guidelines Clearinghouse [68], where clinical guidelines from Australia [69], United States of America [70], Scotland [71], and Canada [72] were found and used as evidence-based general recommendations. Also, elder versions of the above mentioned guidelines were considered, to have a sharper picture of the process evolution (‣ Table 3).

Once the guidelines have been read and understood, the process was defined to comply with the environment of the specialized rehabilitation center (local practices, ‣ Table 3). Thus, textual and graphical clinical pathways used locally were collected and the multidisciplinary rehabilitation team was separately interviewed to gather the requirements, roles, and responsibilities and to identify the main activities of the process. These interviews allowed the analyst to understand and correctly identify all the relevant issues of the environmental context through a model (‣ Figure 1). Interviews’ results showed that the main critical issue was the heterogeneous and unstructured collection of patient’s data that turned into difficulties in data retrieval thus slowing the assessment and evaluation process. Also, patient-reported outcomes measures were collected only on paper-based sheets thus preventing their efficient inclusion in the decision-making process.

The modeling aim was then focused to define the specifications of a new software module for the digital administration and collection of assessment tests to be integrated in the HIS. The graphical UML language was selected as formal notation to simplify the communication between the domain experts and the analysts of the process (R4 in ‣ Table 1) [45].

4.2 Phase 2: Conceptual Modeling

The process designer produced a first high level model of the whole rehabilitation process (Pre-Modeling in ‣ Figure 1) followed by a more detailed one focused only on the assessment of stroke patients (Modeling in ‣ Figure 1); the last sub-step was the validation of the model by the domain experts.

Pre-Modeling

The process participants, along with their main roles were identified and represented using a UML use case diagram (‣ Figure 2). Although different medical specialists were involved in the rehabilitation process, a specialist in physical medicine (the Doctor in ‣ Figure 2) held the major responsibility and was the decision-maker in all process phases. The therapist could be a physiotherapist, a speech or an occupational therapist, and the main task was to perform and supervise the prescribed rehabilitation cycle and to partly help the doctor in the evaluation of patient recovery. The nurses participated in patient enrollment and discharge. The patient was actively involved in both the rehabilitation cycle and the evaluation of his/her recovery.

Fig. 2.

Fig. 2

Pre Modeling: UML Use case diagram of the stroke rehabilitation process.

Figure 3 depicts the high-level UML activity diagram of the process, highlighting the workflow of the main process phases. The first activity was the patient enrollment. It aimed at creating the electronic medical record of the patient in the specialized rehabilitation center and at collecting all the information needed to define the care process. Once the doctor prescribed the rehabilitation program, the rehabilitation cycle started. Then, depending on the results of the monitoring, the doctor had to decide whether the rehabilitation plan was still valid or needed to be modified. The process finished when all the rehabilitation objectives were reached, and the patient was discharged.

Fig. 3.

Fig. 3

Pre Modeling: UML Activity diagram of the stroke rehabilitation process.

A more detailed description of the process phases can be found in [45].

Modeling

The modeling step started from the previously collected information and produced a conceptual model of the specific sub-process according to the adopted formal notation. The sub-process of monitoring patient recovery included the following macro tasks: accessing the software (login), selecting the clinical scale, filling in the clinical scale, the data storage, the data query and the validation of the clinical scale after filling in to check the validity of the responses. The use case diagram in ‣ Figure 4 represents, for each macro task (use case), the actors involved in the execution.

Fig. 4.

Fig. 4

Modeling: UML Use case diagram for filling in the clinical scales.

The schema of a relational database for filling in the clinical scales was defined using the UML class diagram reported in ‣ Figure 5. For each class all the attributes are reported, while methods are not shown for sake of clarity. For each association between classes the name and the cardinality are specified.

Fig. 5.

Fig. 5

Modeling: UML Class diagram for filling in the clinical scales.

A crucial and complex issue was the interaction between the new software artifact, the rehabilitation process, and the HIS. The UML activity diagram in ‣ Figure 6 describes the integration of the administration of assessment tests within the activities of the rehabilitation process, starting from the initial visit and ending with the patient discharge. The main actors involved (reported as headings of the swimlanes) were the patient, the therapist, the doctor and the HIS. Once the rehabilitation was prescribed, the patient underwent the first assessment (using the software at the admission) and the rehabilitation cycle started. Every time an assessment was performed, it consisted of two parallel activities in which both the therapist and the doctor had to use the software to assess the patient using the clinical scales they were responsible for. The data record was shared among the different actors but the software provided a filtered data access depending on the authenticated actor. The grey rectangle in ‣ Figure 6 is described later, because it was introduced as a result of the feedback provided by the domain experts during model validation.

Fig. 6.

Fig. 6

Modeling: UML Activity diagram finalized according to the domain expert feedback suggesting to include patients involved in RCTs. The enrollment of the patient is a pre-condition, and occurred before the definition of rehabilitation program (Figure 3).

Expert Validation

In our use case, the analyst designed the modeled diagrams and tested their correctness and integrity from the syntactic viewpoint. Then, the validated conceptual model was assessed by the staff of the rehabilitation center. One process participant per each actor type took part to this phase. They first evaluated the general adherence of the conceptual model to the process requirements (‣ Table 3); then they assessed the shared protocol described in the model. Each actor gave a positive score both to the adherence and feasibility of the conceptual model. All the expected exceptions were well represented. The main lack found was that the model did not took into the proper account the possibility to assess patients included in the randomized clinical trials (RCT) active in the rehabilitation center. The model was then updated according to this feedback (‣ Figure 6, grey rectangle). Patients eligible for and enrolled in one of the RCT active in the hospital, followed the clinical study protocol that included at least the pre-assessment test, the treatment, and then the post-assessment test (again using the novel software). Once the clinical study was finished, the prescription was updated and the patient could keep going with the rehabilitation cycle till the rehabilitation goals were reached. In this case, the software was used again to assess the patient at discharge. Thus, in the usual rehabilitation delivery, the software was used only at admission and discharge, whereas, if the patient was enrolled in a clinical study, the software system was also used for the assessment required by the RCT protocol. The impact of this deviance affected also the class diagram and in particular the Administration time attribute of the Clinical Scale class.

4.3 Phase 3: Logical Modeling

The logical modeling (phase 3) consisted of mapping the conceptual model to a target executable language. It aimed at translating all the conceptual statements to an executable programming language, so that the instances of the process model could be enacted by suitable software systems. In our case study we designed and developed a software system including a relational database and a graphical user interface used to collect and/or display the results of the clinical assessment. Microsoft Access was chosen as the relational Database Management System (DBMS) and the Structured Query Language (SQL) was adopted for the logical design of the database. While this choice is not the only possible solution, starting from the same conceptual model other logical implementations could be developed.

The final product was presented in [45] and deployed to the rehabilitation center. The rehabilitation team already performed the functional testing of the software and approved the Alpha release of it. A so-called Beta testing phase of the new deployed software is foreseen before a complete integration in the local rehabilitation process.

5. Conclusions

The methodology we have here described through the case study of stroke rehabilitation showed the capability of process modeling to answer actual needs in the clinical practice of the specialized rehabilitation center analyzed. The exemplary modeling effort provided a tool that increased the interactions among actors and the transparency of the actions related to the rehabilitation process, thus answering one of the priorities put forward by the AMA [9]. In fact, all the actors both involved in the modeling effort and later included in its validation took advantage of the graphical representation of the process, and were able to identify their personal activities in it. Moreover, the graphical modeling tool transparently represented all the activities related to rehabilitation, as well as the interactions among actors during the different phases. In this way, all the interactions among the users occurred according to a shared and fully understood protocol (i.e., the model). Dangling situations, expected anomalies, and deadlock were monitored, detected, and properly managed by the process model itself.

For similar reasons, the modeling effort also optimized team-based care [9]: having a shared and precise definition of roles and related privileges/actions would improve team working and delegation, timeliness included. The model described how each actor can perform the actions that require his/her role, thus clarifying the responsibilities and boundaries for each participant involved in the rehabilitation process. In addition, the model was designed to properly monitor also deadlines and delays, thus ensuring a control on the timeliness of care. After a proper field validation, the software developed using the model as specification definition [2], would contribute to improve the continuity of care. The latter is another priority highlighted by the AMA [9]. Then, the software would simplify information collection and retrieval, also avoiding multiple data entry by different actors. The introduction of the software system would also facilitate the outcome analysis and would improve the control of the patients’ clinical pathway in the rehabilitation center. This allows a more efficacious monitoring of patient’s care pathway and a better coordinated work between the units involved, having a more timely access to the data collected for the patient.

The modeling effort used as requirement analysis can be extended to clinical scales and assessment measures used in other rehabilitation processes, a part from stroke rehabilitation, facilitating the improvement of other hospital processes. Moreover, the high-level definition of the process phases provides a way to identify software modules needed in each phase, thus helping in achieving, especially from the non-technical users’ viewpoint, a sharper picture of the software product modularity [9], to guide the choice/implementation of better hospital information systems and Electronic Health Records (EHRs). The modeling effort was also able to clearly represent the information exchanged among phases, actors, and actions.

Finally, the case study here reported can be enriched by including a more detailed description of all the exceptions that can occur during process execution, and through the definition of check points through which raise alerts in the case of deviations from the shared protocol.

The methodology here proposed can be used to guide process modeling, independently from the specific medical domain in which the modeling effort is done. The methodology benefits not only during information system design (or even re-engineering), but also during clinical pathway design, analysis, and validation. Major advantages are in managing the complexity of the process, as well as ensuring transparency of activities among the involved actors. Future studies should evaluate the implementation of process models describing the clinical behavior and the actual patient outcomes to provide evidence of their cost-effectiveness. As for the patient outcomes, some metrics such as the length of stay, the health-related quality of life, and the functional independence should be used to assess process quality. For instance, an approach to evaluate the training intervention for implementing the process is the Kirkpatrick’s Hierarchy [73]. It includes four levels, reaction, behavior, learning, and results. Particularly, the results level means that the expectations about introducing the model have been thoroughly satisfied.

Acknowledgements

None

Footnotes

Clinical Relevance Statement

Process modeling gives the methodological basis for quality improvement in medicine because it is able to break out complexity and provide transparency to medical processes. Indeed, modeling healthcare processes can
  1. help in establishing shared protocols for patient’s care; thus facilitating the adherence to the shared protocols and guidelines, and limiting problems due to incomplete communication or misunderstandings among different actors;
  2. increase patient’s safety by means of a continuous monitoring of deviances from the protocols, redundancies, and failures, thus early identifying problems that could lead to un-prevented errors;
  3. provide to all of the actors involved a fully understanding of the information flow, thus identifying requirements and specifications for information system re-engineering and interoperability;
  4. detecting process weaknesses thus designing corrective measures;
  5. optimizing the use of resources.

Statement on Conflicts of Interest The authors declare that they have no conflicts of interest in the research.

Human Subject Research Approval Human and/or animal subjects were not included in the work.

References

  • 1.Ruiz F, García F, Calahorra L, Llorente C, Gonçalves L, Daniel C, Blobel B. Business Process Modeling in Healthcare. Stud Health Technol Inform 2012; 179: 75–87. [PubMed] [Google Scholar]
  • 2.Garde S, Knaup P. Requirements engineering in health care: the example of chemotherapy planning in paediatric oncology. Requirements eng 2006; 11(4):265–278. [Google Scholar]
  • 3.Baacke L, Mettler T, Rohner P. Component-based process modelling in health care. Proc in the 17th European Conference on Information Systems ECIS Verona, Italy 2009; 430–441. [Google Scholar]
  • 4.Workflow Management Coalition Specification, Workflow Management Coalition, Terminology & Glossary (Document No. WFMC-TC-1011). Workflow Management Coalition Specification 1999. [Google Scholar]
  • 5.Chiao CM, Künzle V, Reichert M. Integrated modeling of process- and data-centric software systems with PHILharmonicFlows. in CPSM@ICSM 2013; 1–10. [Google Scholar]
  • 6.Horsky J, Gutnik L, Patel VL. Technology for emergency care: cognitive and workflow considerations. AMIA Annu Symp Proc 2006; 344–348. [PMC free article] [PubMed] [Google Scholar]
  • 7.Risser DT, Rice MM, Salisbury ML, Simon R, Jay GD, Berns SD. The potential for improved teamwork to reduce medical errors in the emergency department. The MedTeams Research Consortium. Ann Emerg Med 1999; 34(3):373–383. [DOI] [PubMed] [Google Scholar]
  • 8.Ozkaynak M, Brennan P. An observation tool for studying patient-oriented workflow in hospital emergency departments. Methods Inf Med 2013; 52(6):503–513. [DOI] [PubMed] [Google Scholar]
  • 9.American Medical Association [Internet], Chicago: The Association; c1995–2015 [updated 2014 Sep 16; cited 2015 Nov 5]. 8 top challenges and solutions for making EHRs usable; [about 2 screens]. Available from: http://www.ama-assn.org/ama/pub/ama-wire/ama-wire/post/8-top-challenges-solutions-makingehrs-usable.
  • 10.Barbarito F, Pinciroli F, Mason J, Marceglia S, Mazzola L, Bonacina S. Implementing standards for the interoperability among healthcare providers in the public regionalized Healthcare Information System of the Lombardy Region. J Biomed Inform 2012; 45(4):736–745. [DOI] [PubMed] [Google Scholar]
  • 11.Malhotra S, Jordan DA, Shortliffe EH, Patel VL. Workflow modeling in critical care: Piecing together your own puzzle. J Biomed Inform 2007; 40(2):81–92. [DOI] [PubMed] [Google Scholar]
  • 12.Schweitzer M, Lasierra N, Oberbichler S, Toma I, Fensel A, Hoerbst A. Structuring clinical workflows for diabetes care: an overview of the OntoHealth approach. Appl Clin Inform 2014; 5(2):512–526. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Kumarapeli P, De Lusignan S, Ellis T, Jones B. Using Unified Modelling Language (UML) as a process-modelling technique for clinical-research process improvement. Med Inform Internet Med 2007; 32(1):51–64. [DOI] [PubMed] [Google Scholar]
  • 14.de Lusignan S, Krause P, Michalakidis G, Vicente MT, Thompson S, McGilchrist M, Sullivan F, van Royen P, Agreus L, Desombre T, Taweel A, Delaney B. Business Process Modelling is an Essential Part of a Requirements Analysis. Contribution of EFMI Primary Care Working Group. Yearb Med Inform, 2012; 7(1):34–43. [PubMed] [Google Scholar]
  • 15.Vawdrey DK, Wilcox LG, Collins S, Feiner S, Mamykina O, Stein DM, Bakken S, Fred MR, Stetson PD. Awareness of the Care Team in Electronic Health Records. Appl Clin Inform 2011; 2(4):395–405. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Lenz R, Reichert M. IT support for healthcare processes – premises, challenges, perspectives. Data Knowl. Eng. 2007; 61(1):39–58. [Google Scholar]
  • 17.Marceglia S, Mazzola L, Bonacina S, Tarquini P, Donzelli P, Pinciroli F. A Comprehensive e-prescribing model to allow representing, comparing, and analyzing available systems. Methods Inf Med 2013; 52(3):199–219. [DOI] [PubMed] [Google Scholar]
  • 18.Howard RMT, Barrows S. Problem-Based Learning: An Approach to Medical Education (Springer Series on Medical Education). Springer Publishing Company. [Google Scholar]
  • 19.Oshima Lee E, Emanuel EJ. Shared decision making to improve care and reduce costs. N Engl J Med 2013; 368(1):6–8. [DOI] [PubMed] [Google Scholar]
  • 20.Panzarasa S, Maddè S, Quaglini S, Pistarini C, Stefanelli M. Evidence-based careflow management systems: the case of post-stroke rehabilitation. J Biomed Inform 2002; 35(2):123–139. [DOI] [PubMed] [Google Scholar]
  • 21.Huang B, Zhu P, Wu C. Customer-centered careflow modeling based on guidelines. J Med Syst 2012; 36(5):3307–3319. [DOI] [PubMed] [Google Scholar]
  • 22.Reichert M. What BPM Technology Can Do for Healthcare Process Support. in AIME, 2011; 2–13. [Google Scholar]
  • 23.Baresi L, Casati F, Castano L, Fugini M, Grefen P, Mirbel I, Pernici B, Pozzi G. Workflow Design Methodology. In Database Support for Workflow Management The WIDE Project. 491, Paul Grefen, Barbara Pernici, Gabriel Sánchez, Jochem Vonk, Erik Boertjes, 1999. [Google Scholar]
  • 24.Unertl KM, Weinger MB, Johnson KB, Lorenzi NM. Describing and Modeling Workflow and Information Flow in Chronic Disease Care. J Am Med Inform Assoc 2009; 16(6):826–883. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Barbarito F, Pinciroli F, Barone A, Pizzo F, Ranza R, Mason J, Mazzola L, Bonacina S, Marceglia S. Implementing the lifelong personal health record in a regionalised health information system: The case of Lombardy, Italy. Comput Biol Med 2015; 59: 164-174. [DOI] [PubMed] [Google Scholar]
  • 26.Peleg M, Somekh J, Dori D. A methodology for eliciting and modeling exceptions. J Biomed Inform, 2009; 42(4):736–747. [DOI] [PubMed] [Google Scholar]
  • 27.Rebuge Á, Ferreira DR. Business process analysis in healthcare environments: A methodology based on process mining. Inf Syst 2012; 37(2):99–116. [Google Scholar]
  • 28.Huang Z, Lu X, Duan H. On mining clinical pathway patterns from medical behaviors. Artificial Intelligence in Medicine 2012; 56(1):35–50. [DOI] [PubMed] [Google Scholar]
  • 29.Bonacina S, Marceglia S, Pinciroli F. Barriers Against Adoption of Electronic Health Record in Italy. J Healthc Eng 2011; 2(4):509–526. [Google Scholar]
  • 30.Kotronoulas G, Kearney N, Maguire R, Harrow A, Di Domenico D, Croy S, MacGillivray S. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol 2014; 32(14):1480–1501. [DOI] [PubMed] [Google Scholar]
  • 31.Koh HK, Brach C, Harris LM, Parchman ML. A proposed ‘Health Literate Care Model’ would constitute a systems approach to improving patients’ engagement in care. Health Affairs 2013; 32(2):357–367. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Anzböck R, Dustdar S. Modeling and implementing medical Web services. Data Knowl Eng 2005; 55(2):203–236. [Google Scholar]
  • 33.Leonardi G, Panzarasa S, Quaglini S, Stefanelli M, van der Aalst WMP. Interacting agents through a web-based health serviceflow management system. Journal of Biomedical Informatics 2007; 40(5):486–499. [DOI] [PubMed] [Google Scholar]
  • 34.Reichert M, Weber B. Enabling Flexibility in Process-Aware Information Systems. Springer, 2012. [Google Scholar]
  • 35.Peleg M, Tu SW. Design patterns for clinical guidelines. Artificial Intelligence in Medicine 2009; 47(1):1–24. [DOI] [PubMed] [Google Scholar]
  • 36.González-Ferrer A, ten Teije A, Fdez-Olivares J, Milian K. Automated generation of patient-tailored electronic care pathways by translating computer-interpretable guidelines into hierarchical task networks. Artif Intell Med 2013; 57(2):91–109. [DOI] [PubMed] [Google Scholar]
  • 37.Musen MA, Tu SW, Das AK, Shahar Y. EON: a component-based approach to automation of protocol-directed therapy. J Am Med Inform Assoc 1996; 3(6):367–388. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.De Clercq PA, Blom JA, Hasman A, Korsten HH. GASTON: an architecture for the acquisition and execution of clinical guideline-application tasks. Med Inform Internet Med 2000; 25(4):247–263. [DOI] [PubMed] [Google Scholar]
  • 39.Terenziani P, Molino G, Torchio M. A modular approach for representing and executing clinical guidelines. Artif Intell Med 2001; 23(3):249–276. [DOI] [PubMed] [Google Scholar]
  • 40.Peleg M, Boxwala AA, Bernstam E, Tu S, Greenes RA, Shortliffe EH. Sharable representation of clinical guidelines in GLIF: relationship to the Arden Syntax. J Biomed Inform 2001; 34(3):170–181. [DOI] [PubMed] [Google Scholar]
  • 41.Quaglini S, Stefanelli M, Lanzola G, Caporusso V, Panzarasa S. Flexible guideline-based patient careflow systems. Artif Intell Med 2001; 22(1):65–80. [DOI] [PubMed] [Google Scholar]
  • 42.Klein M, Dellarocas C. A Knowledge-based Approach to Handling Exceptions in Workflow Systems. Computer Supported Cooperative Work 2000; 9 (3/4): 399–412. [Google Scholar]
  • 43.de Carvalho ECA, Jayanti MK, Batilana AP, Kozan AMO, Rodrigues MJ, Shah J, Loures MR, Patil S, Payne P, Pietrobon R. Standardizing clinical trials workflow representation in UML for international site comparison. PLoS ONE 2010; 5(11): e13893. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Assimakopoulos NA. Workflow management with systems approach: anticipated and ad-hoc workflow for scientific applications ISA Trans 2000; 39(2):153–167. [DOI] [PubMed] [Google Scholar]
  • 45.Ferrante S, Bonacina S, Pinciroli F. Modeling stroke rehabilitation processes using the Unified Modeling Language (UML). Comp in Bio and Med 2013; 43(10):1390–1401. [DOI] [PubMed] [Google Scholar]
  • 46.Garde S, Baumgarten B, Basu O, Graf N, Haux R, Herold R, Kutscha U, Schilling F, Selle B, Spiess C, Wetter T, Knaup P. A meta-model of chemotherapy planning in the multi-hospital/multi-trial-center-environment of pediatric oncology. Methods Inf Med 2004; 43(2):171–183. [PubMed] [Google Scholar]
  • 47.Shiki N, Ohno Y, Fujii A, Murata T, Matsumura Y. Time process study with UML. Methods Inf Med 2009; 48(6):582–588. [DOI] [PubMed] [Google Scholar]
  • 48.van der Aalst W, ter Hofstedeb A. YAWL – Yet Another Workflow Language. Information Systems. Information Systems 2005; 30: 245–275. [Google Scholar]
  • 49.Reichert M, Rinderle S, Almagro PL. ADEPT flex – Supporting Dynamic Changes of Workflows Without Loosing Control. Journal of Intelligent Information Systems 1998; 10(2):93–129. [Google Scholar]
  • 50.Reichert M, Rinderle S, Dadam P. ADEPT Workflow Management System: Flexible Support for Enterprise-Wide Business Processes. in Proceedings of the international conference on Business process management 2003; 370–379. [Google Scholar]
  • 51.Fasola G, Rizzato S, Merlo V, Aita M, Ceschia T, Giacomuzzi F, Lugatti E, Meduri S, Morelli A, Rocco M, Tozzi V. Adopting integrated care pathways in non-small-cell lung cancer: from theory to practice. J Thorac Oncol 2012; 7(8):1283–1290. [DOI] [PubMed] [Google Scholar]
  • 52.Fasola G, Aprile G, Aita M. A model to estimate human resource needs for the treatment of outpatients with cancer. J Oncol Pract 2012; 8(1):13–17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Shahar Y, Miksch S, Johnson P. The Asgaard project: a task-specific framework for the application and critiquing of time-oriented clinical guidelines. Artif Intell Med 1998; 14(1–2): 29–51. [DOI] [PubMed] [Google Scholar]
  • 54.Mendling J. Metrics for Process Models: Empirical Foundations of Verification, Error Prediction, and Guidelines for Correctness 2008; 6, Springer. [Google Scholar]
  • 55.Bertino E, Ferrari E, Atluri V. The Specification and Enforcement of Authorization Constraints in Workflow Management Systems. ACM Trans Inf Syst Secur 1999; 2(1):65–104. [Google Scholar]
  • 56.Combi C, Degani S. Seamless (and Temporal) Conceptual Modeling of Business Process Information. in ADBIS 2011; 2: 23–32. [Google Scholar]
  • 57.Tu SW, Campbell JR, Glasgow J, Nyman MA, McClure R, McClay J, Parker C, Hrabak KM, Berg D, Weida T, Mansfield JG, Musen MA, Abarbanel RM. The SAGE Guideline Model: achievements and overview. J Am Med Inform Assoc 2007; 14(5):589–598. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58.Swartz J. Airport 95: Automated Baggage System?. Software Engineering Notes 1996; 21(2):79–83. [Google Scholar]
  • 59.Leape LL, Lawthers AG, Brennan TA, Johnson WG. Preventing medical injury. QRB Qual Rev Bull 1993; 19(5):144–149. [DOI] [PubMed] [Google Scholar]
  • 60.Liaw ST, Deveny E, Morrison I, Lewis B. Clinical, information and business process modeling to promote development of safe and flexible software. Health Informatics J 2006; 12(3):199–211. [DOI] [PubMed] [Google Scholar]
  • 61.Business Process Modeling Language. Business Process Management Institute; 2003. [Google Scholar]
  • 62.Askari M, Westerhof R, Eslami S, Medlock S, de Rooij SE, Abu-Hanna A. A combined disease management and process modeling approach for assessing and improving care processes: A fall management case-study. I J Medical Informatics 2013; 82(10):1022–1033. [DOI] [PubMed] [Google Scholar]
  • 63.Jun GT, Ward J, Morris Z, Clarkson J. Health care process modelling: which method when?. International Journal for Quality in Health Care 2009; 21(3):214–224. [DOI] [PubMed] [Google Scholar]
  • 64.Geambasu CV. BPMN vs. UML Activity Diagram for Business Process Modeling. Journal of Accounting and Management Information Systems 2012; 11(4):637–651. [Google Scholar]
  • 65.Rumbaugh JE, Jacobson I, Booch G. The unified modeling language reference manual. Addison-Wesley-Longman, 1999. [Google Scholar]
  • 66.OMG-UML-SUP. 2005. Unified modeling language: Superstructure. OMG document formal/05–07–04. [Google Scholar]
  • 67.Shiki N, Ohno Y, Fujii A, Murata T, Matsumura Y. Unified Modeling Language (UML) for hospital-based cancer registration processes. Asian Pac J Cancer Prev 2008; 9(4):789–796. [PubMed] [Google Scholar]
  • 68.National Guidelines Clearinghouse [Internet]. Rockville, MD: Agency for Healthcare and Quality; c2000–01 [updated 2015 June 18; cited 2015 Nov 5]. Available from: http://www.guideline.gov/. [Google Scholar]
  • 69.National Stroke Foundation, Australia. Clinical Guidelines for Stroke Management 2010 – Recommendations. 2010. [Google Scholar]
  • 70.Bates B, Choi JY, Duncan PW, Glasberg JJ, Graham GD, Katz RC, Lamberty K, Reker D, Zorowitz RUS Department of Defense; Department of Veterans Affairs. Veterans Affairs/Department of Defense Clinical Practice Guideline for the Management of Adult Stroke Rehabilitation Care: Executive summary. Stroke 2005; 36(9):2049–2056. [DOI] [PubMed] [Google Scholar]
  • 71.Scottish Intercollegiate Guidelines Network. Management of Patients with Stroke or TIA: Assessment, Investigation, Immediate Management and Secondary Prevention. A national clinical guideline. 2008. [Google Scholar]
  • 72.Lindsay P, Bayley M, McDonald A, Graham ID, Warner G, Phillips S. Toward a more effective approach to stroke: Canadian Best Practice Recommendations for Stroke Care. Canadian Medical Association Journal 2008; 178(11):1418–1425. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. 2006; 3rd ed. Berkeley: Berrett-Koehler Publishers, Inc. [Google Scholar]

Articles from Applied Clinical Informatics are provided here courtesy of Thieme Medical Publishers

RESOURCES