Evaluation of System Mapping Approaches in Identifying Patient Safety Risks

Abstract

Objective

While many system mapping approaches (SMAs) have been broadly used in safety-critical industries, few have so far been employed in the healthcare field to assist in the identification of patient safety risks. In this study, we evaluated a set of system modelling approaches to assess their potential contribution to the identification of risks affecting patient safety. The aim was to gain a greater understanding of the practical application of system modelling approaches with the help of the risk categorisation framework developed in this study.

Setting

We conducted this study in a newly established Adult Attention Deficit Hyperactivity Disorder (ADHD) service at Cambridge and Peterborough Foundation Trust.

Study Participants

Eight key stakeholders of the chosen service, including clinicians, managers, and administrative staff, were individually asked to evaluate a set of pre-defined six SMAs according to their usefulness in identifying patient safety risks through interview-based questionnaires.

Results

It was found that each SMA could be useful in the chosen healthcare service in different ways. Further, specific types of diagrams were selected by stakeholders as more useful than others in identifying different sources of risks within the given system.

Conclusions

The results of the evaluation showed that the system diagram is the most useful SMA in risk identification within the given system, while limited time, resources, and experience of stakeholders with SMAs may present possible obstacles for their potential use in the healthcare field in future.

1. Introduction

Since the publication of the Institute of Medicine (IOM) report, To Err Is Human: Building a Safer Health System [1], various studies have been conducted and indicated that the rate of harm remains high; thousands of people around the world are harmed every year [2–4]. The result of high frequency of medical errors suggested that further research and improvements are needed to improve health services and make health systems safer for patients [5].

As one of the suggestions to improve patient safety, risk identification has been proposed within the scope of risk management to investigate actual and potential harm. Using various tools and methods, robust risk identification aims to identify a comprehensive lists of risks, since no actions may be taken to avoid or diminish the effects of unidentified hazards [6,7].

In order to better understand the risks in any systems, information and experience need to be available through different inputs, such as (1) knowledge of people, (2) incident reports, and (3) system mapping approaches (SMAs), also known as process maps, process models, and diagrams [8]]. In order to define a generic risk identification process, Figure 1 proved useful in showing the range of inputs and outputs; while the inputs are based on the National Offshore Petroleum Safety Authority (NOPSA) guidance [8], the risk outputs are based on the definitions given by International Organization for Standardization (ISO 31000) [9] and Ericson [10].

Figure 1. Risk identification process.

Figure 1 shows how different inputs, treated using a range of retrospective and prospective methods can help identify risks as outputs. Further, their combination may clearly provide a better result in terms of determining a comprehensive list of risks in selected systems.

Recent research showed that knowledge of people and incident reports added value to the overall risk identification process by adding different sources of risks, but none provided a comprehensive picture about all possible risks in the chosen healthcare setting [11]. The study therefore suggested for future study that applying SMAs and combining them with other two inputs might provide better results in risk identification. However, there is no clear evidence regarding what types of risks can be identified using different SMAs; their contribution to risk identification is thus still not very clear.

Moreover, in another recent study, it was shown that SMAs have not been introduced in organisation-level risk management guidance in National Health Service (NHS) settings [12]. While other safety-critical industries, such as chemical and aviation industries, use strategies to select appropriate SMAs, there is no clear evidence paralleling these practices in the healthcare field, particularly in the patient safety context.

The accurate overall representation of a system is important in attempting to truly cover each element in the system. SMAs can therefore provide good system representation to systematically analyse the system in question [13]. For instance, Redmill et al. [14] highlighted that visualization allows workers to easily observe a system and understand how it works. The use of visual maps, therefore, can also aid in the identification of the critical elements of the given system [15].

They can be used independently in risk identification, and they can also be used together with Prospective Hazard Analysis (PHA) methods which have recently gained awareness in healthcare to address the system-level determinants of patient safety [16]. For instance, PHA methods, such as Failure Modes and Effect Analysis (FMEA), Hazards and Operability (HAZOP), and Structured What-if Techniques (SWIFT), have tabular structures. Hence, visualization of healthcare systems through SMAs can potentially strengthen their risk identification process. In other safety-critical industries, many types of SMAs have been used for risk identification along with different PHA methods. For instance, HAZOP uses process flow diagrams and data flow diagrams to define material and information flow between system components and operations, while FMEA uses flow diagrams, functional block diagrams, and reliability block diagrams to define system components and functions. In some cases, multiple kinds of diagrams have been used, to enhance the understanding of the system from different perspectives [13].

While a wide range of SMAs have been used in other safety-critical industries, such as the chemical industry, only few have been employed in the healthcare field to assist in the identification of risks [13,17]. Further, it has also been found that the use of multiple SMAs is limited in the NHS; in most cases, flowcharts and hierarchical task analysis diagrams have been preferred [13,18]. Although various SMAs have the potential to generate clear descriptions of the wide range of processes employed in the healthcare field, time pressure can limit the number of SMA types that can be utilised [18].

In order to take full advantage of using SMAs in healthcare, Jun [13] carried out a comprehensive study concerned solely with the applicability of SMAs to healthcare for patient safety. Because each SMA has its own set of notations and layouts, Jun emphasised the need to use the correct map for the analysis of a given healthcare context, and identified and characterized various types of diagrams applicable to the design of healthcare delivery systems [13,17]. The research also showed that other SMAs were helpful in understanding the diversity and complexity of the given processes. For instance, communication diagrams helped participants understand interactions in the system. While swim-lane activity diagrams helped participants understand the roles and responsibilities in a given system, state transition diagrams were helpful from a patient-centred perspective [17]. In another research, Jun and his colleagues [19] also provided an indication regarding the potential use of SMAs in identifying different types of hazards, such as task-related, human-resource related and information/material-related hazards.

Another comprehensive study on SMAs was carried out in the development of the PHA toolkit developed by Clarkson et al. [20]. In this research, a number of SMAs were reviewed to consider their practical applicability in the healthcare context, and also to understand their possible associations with particular PHA methods. Regarding the review results, six distinct SMAs were shortlisted in the PHA toolkit, and identified as useful for providing fundamental diagrammatic representations to capture a range of system attributes in prospective hazard analysis applications. Each diagram shortlisted in the PHA Toolkit can be briefly described as follows:

• 1-Task diagrams describe a hierarchy of operations and plans
• 2-Information diagrams describe a hierarchy of information and/or material
• 3-Organisational diagrams describe a hierarchy of people and/or roles within organisation(s)
• 4-System diagrams represent how data are transferred through activities
• 5-Flow diagrams represent activities occurring in sequence or in parallel
• 6-Communication diagrams represent information and material flows between people and process

While the diagrams can help identify risks in a chosen healthcare system, it can also be said that each diagram may help identify a particular type of risk since each represents the system from different perspectives. It is therefore imperative to understand the usefulness of each SMA in the identification of particular risk source within a system.

There are various frameworks available in patient-safety context representing various error-producing conditions and contributory factors that can help categorise different risk sources [21–24]. The ultimate aim of such frameworks is to help understand healthcare systems by considering various system components to provide a comprehensive system view. Therefore such classifications can be useful in risk identification and potentially utilized as a reference for possible categorisation of risk sources leading to situations harmful to patients. In this study, we therefore aimed to evaluate a range of SMAs to address their potential contribution in identifying different risk sources in patient safety context.

2. Methods

2.1. Data Collection

We first aimed to develop a risk identification framework to include various SMAs as inputs, and risk sources as outputs within the risk identification process. In the framework, we included six different SMAs as inputs, as shortlisted in the PHA Toolkit [20]. As mentioned earlier, there are a number of classifications that can be utilised as a reference to categorise risk sources. Although components in each framework is very similar to each other, we preferred to use the NPSA (National Patient Safety Agency) contributing factors classification framework used originally in incident reporting and root cause analysis within the NHS [25]. Since this study was conducted within the NHS area, study participants would be more familiar with the content of the NPSA framework compared to any other frameworks.

In an earlier study involved the NPSA framework [11], difficulties were experienced in differentiating the contributing factors as distinct risk sources. To produce an accurate categorisation, some factors were considered subordinate to other types of risk sources. We thereby developed a risk identification process that includes six types of SMAs as inputs, and seven types of risk sources as outputs, as shown in Figure 2.

Figure 2. Risk identification process with shortlisted SMAs (inputs) and risk sources (outputs).

Throughout the framework, SMAs were evaluated at the newly established Adult Attention Deficit Hyperactivity Disorder (ADHD) Service at Cambridge and Peterborough Foundation Trust. The Adult ADHD Service cares for young people experiencing significant impairment as a result of ADHD after the age of 17. The service aims to provide specialist diagnostic services and a range of pharmacological and psychosocial interventions for those with adult ADHD, to the adult population of Cambridgeshire and Peterborough. These services are delivered by a multidisciplinary team of professionals, such as psychologists, nurses and admin staff, led by a consulting psychiatrist.

Throughout the evaluation of SMAs, two data sets were obtained to help determine the usefulness of each SMA in the identification of patient safety risks at the ADHD service.

The first data set was gathered to obtain initial data about the adult ADHD Service. This information was collected prior to the application and evaluation of SMAs. In order to build SMAs, initial data were obtained from the service leader; as the key participant in this study, the service leader provided all inputs needed for the research, in a number of face-to-face meetings. The internal procedural documents were also helpful to understand the service. Through the use of the Cambridge Advanced Modeller (CAM) tool [26], six types of SMAs were drawn up (see Appendix). When these were accomplished, the service leader was asked to confirm the suitability of the content of the SMAs with the real practice. Then we made further arrangements to conduct evaluation sessions with each stakeholder working in the chosen mental health service.

Table 1 shows the job titles of the participants, and provides further details on their experience in the NHS, their familiarity and experience in using SMAs.

Table 1. Participants in SMA evaluation.

No.	Job title	Experience in the NHS	Familiarity with SMAs	Experience and Purpose in using SMAs
1	Service Manager	28 years	Medium	A couple of times - Risk assessment, operational policies, communication
2	Consultant Psychiatrist	24 years	Very familiar	A couple of times - System understanding and communication
3	Specialist R. Psychiatrist	10 years	Medium	Sometimes - Clinical training purposes
4	Admin Support	3.5 years	Not familiar at all	None
5	Clinical Psychologist	13 years	Not familiar at all	None
6	Clinical Psychologist	5 years	Not familiar at all	None
7	Specialty Registrar	12 years	Medium	None
8	Nurse Specialist	24 years	Medium	None

The second data set was collected to determine the potential contribution of SMAs to risk identification via individual-based workshops and questionnaires with each participant. The participants were then asked to indicate their level of agreement to eight statements (see Table 2) for supporting the use of SMAs in risk identification. Then the rankings were expressed numerically; to aid numerical analysis after all the data were collected. Five-point scales (poor: 1 – excellent: 5) were used and supported, using numerical responses to aid in numerical analysis. The participants were asked to tick the one option that most closely characterised their opinion, according to their experience with each SMA. Since we have small data sets, a test of significance for normality may lack power to detect the deviation of the variable from normality. In this study, we therefore provided only descriptive statistics, such as mean (μ), median and interquartile range.

Table 2. Summary Statistics of SMA Evaluation (n = 8).

Statement on Diagram’s Usefulness in Risk Identification within the System	System Mapping Approaches (SMAs)
	Organisational Diagram				Information Diagram				Task Diagram				Flow Diagram				Communication Diagram				System Diagram
	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR	Mean, µ	Standard Deviation	Median	Inter Quartile Range, IQR
1. General usefulness in risk identification	2.8	0.7	3.0	1.0	3.9	0.4	4.0	0.0	3.3	0.5	3.0	0.3	4.1	0.4	4.0	0.0	3.4	0.7	3.5	1.0	4.5	0.5	4.5	1.0
1.1 Particular usefulness in identifying task-related risks	1.8	0.7	2.0	1.0	2.8	0.9	2.5	1.3	3.3	0.7	3.0	1.0	3.5	0.5	3.5	1.0	2.8	0.9	2.5	1.3	3.8	0.5	4.0	0.3
1.2 Particular usefulness in identifying equipment-related risks	1.3	0.5	1.0	0.3	2.6	1.1	2.5	1.3	1.6	0.5	2.0	1.0	1.6	0.5	2.0	1.0	1.6	0.5	2.0	1.0	2.3	1.0	2.0	1.3
1.3 Particular usefulness in identifying organisational risks	2.5	0.5	2.5	1.0	2.4	0.7	2.5	1.0	2.4	0.7	2.5	1.0	3.1	0.8	3.0	1.3	2.8	0.9	2.5	1.3	3.4	0.7	3.5	1.0
1.4 Particular usefulness in identifying environmental risks	1.5	0.5	1.5	1.0	1.5	0.5	1.5	1.0	1.5	0.5	1.5	1.0	2.8	0.9	2.5	1.3	2.3	1.0	2.0	1.3	2.8	0.9	2.5	1.3
1.5 Particular usefulness in identifying communication risks	2.3	1.0	2.0	1.3	3.0	0.9	3.0	2.0	2.8	0.9	2.5	1.3	3.4	0.7	3.5	1.0	3.3	0.7	3.0	1.0	3.0	0.5	3.0	0.0
1.6 Particular usefulness in identifying staff-related risks	2.8	0.9	2.5	1.3	1.9	1.0	2.0	1.0	1.9	1.0	2.0	1.0	2.5	0.5	2.5	1.0	2.1	0.6	2.0	0.3	2.3	1.0	2.0	1.3
1.7 Particular usefulness in identifying patient-related risks	1.6	0.5	2.0	1.0	3.3	0.7	3.0	1.0	1.9	1.0	2.0	1.0	2.1	0.6	2.0	0.3	2.1	0.6	2.0	0.3	2.3	1.0	2.0	1.3

3. Results

During this study, eight participants individually evaluated six different diagrams. The statements and the summary statistics of the SMA evaluation are shown in Table 2 below:

The results shows that the participants rated each diagram type as significantly similar, in terms of support for their use in risk identification. It was therefore possible to analyse the results of the participants’ ratings collectively. For the first statement — “The diagram is useful in identifying risks within the system.” — participants agreed that the most helpful diagram in risk identification is the system diagram, (μ = 4.5). The system diagram was followed by the flow diagram (μ = 4.1). The least useful diagram was found as the organisational diagram (μ = 2.8).

With respect to the types of risk sources, it was found that some participants were familiar with such classifications from incident reporting and root cause analysis. Hence, it helped participants make correlations between SMAs and different sources of risks. It appeared that different diagram types are more useful in identifying different sources of risks. For instance, the participants agreed that that information diagram is the most useful diagram type (μ = 3.3) in identifying patient-related risks, while the flow diagram was the most useful diagram type in identifying communication risks (μ = 3.4).

4. Discussion

Results indicated that each SMA has different strengths and limitations in identifying different types of risk sources. Therefore, depending on the nature of the system chosen, particular SMA can be selected to provide a more comprehensive risk identification. For instance, if a chosen system has mainly environmental risk sources in its content, a flow diagram or a system diagram would be more useful than any other type of SMAs. As a result of this study, the suitability of different SMAs can be illustrated, as shown in Table 3, in order to identify a particular range of risk sources. In this table, the diagrams are marked with two different sizes of tick marks (bigger mark means more useful) to show their usefulness, based on their average rates.

Table 3. SMAs and characteristics of risk sources.

SMA type	Organisational	Information	Task	Flow	Communication	System
Risk Sources
Task-related risks			✓	✔		✔
Environmental risks				✔	✓	✔
Equipment-related risks		✔				✓
Communication risks		✓		✔	✔	✓
Organisational risks				✔	✓	✔
Staff-sourced risks	✔			✓		✓
Patient-sourced risks		✔				✓

Table 3 shows the suitability of different SMAs for prioritisation according to their ability to identify a particular risk sources.

As with earlier studies [19,27], each type of SMA was important, as each influenced the representation of the chosen healthcare service. It can therefore be concluded that each approach can identify different risk sources. As also suggested in earlier studies [27], multiple SMAs can be used to ensure that all risk sources are captured; therefore a better risk picture can be obtained. However, this is controversial and unrealistic, perhaps even a less effective approach, since lack of time and resources always constitutes a pressure in healthcare settings. To be pragmatic regarding the availability of current resources in healthcare, it should be noted that often only one SMA can be used. Therefore, it is always useful to know the most suitable SMA during the application of risk identification and further risk assessment. To address this, our study provided valuable insights, at least at the outset, to help understand which SMA is most useful with regard to the risk sources that can affect the chosen service more than others.

In this study, a number of criteria are therefore suggested to select the most appropriate SMA, depending on the nature of the healthcare system. For instance, the rates of risk sources and incident types occurring in the past can guide healthcare organisations in selecting the most common risk sources in a given healthcare system. The main sources of risk in the system can be captured via the results of incident investigations; the most appropriate SMA can therefore be selected with regard to the main risk source identified in the chosen healthcare service. This is one of the most significant outcomes of this study — it can help establish a bridge between retrospective results (e.g. incident types) and SMAs. There would, of course, be no retrospective results available with new services, as observed in this study. In such circumstances, stakeholders of the system can evaluate the system to select the most common risk sources. Therefore, the most appropriate SMA can be chosen to use in risk identification practices.

While the evaluation workshops were completed successfully, a number of issues arose due to the complexity of some of the SMAs. As shown earlier in Table 1, participants had only limited familiarity and experience in using SMAs. Even when they had been given instructions by the facilitator, they sometimes found the use of diagrams redundant, as well as too complex. For instance, throughout the SMA evaluation, most participants asked the meaning of the basic nodes, links, and functions used in the content of SMAs, even though the facilitator had given brief instructions earlier. It can be assumed, therefore, that detailed instructions and training may be needed in future research to get the greatest benefit from the use of SMAs in identifying patient safety risks. Moreover, it can be addressed that from just eight participants looking at six diagrams, it is not easy to confirm what can be inferred quantitatively. However, qualitative results serve as reliable indicators to consider in current risk identification practice and future research.

5. Conclusions

In this study, it was noted that a number of risk sources interacted in the system, and may have influenced the outcome of care delivered. It is therefore important to know how fully systems and their components are described; their interaction is understood and applied in a dynamic and complex healthcare systems. Throughout the evaluation in this study, SMAs were found to be useful in describing the system and identifying potential patient safety risks for the chosen healthcare service. Further, different types of diagrams were found to be more useful than others in identifying particular risk sources within the given system, as shown in Table 3. While the evaluation showed that the system diagram was the most useful SMA in risk identification, it can also be suggested that amalgamation of the use of multiple maps can help identify a comprehensive list of risks; however, it may not always be feasible to use all SMAs in the risk identification practice, due to limited resources, such as limited time, financial resources, and experience of system users with SMAs.