An Analytic Framework to Assess Organizational Resilience

Abstract

Background

Resilience engineering is a paradigm for safety management that focuses on coping with complexity to achieve success, even considering several conflicting goals. Modern sociotechnical systems have to be resilient to comply with the variability of everyday activities, the tight-coupled and underspecified nature of work, and the nonlinear interactions among agents. At organizational level, resilience can be described as a combination of four cornerstones: monitoring, responding, learning, and anticipating.

Methods

Starting from these four categories, this article aims at defining a semiquantitative analytic framework to measure organizational resilience in complex sociotechnical systems, combining the resilience analysis grid and the analytic hierarchy process.

Results

This article presents an approach for defining resilience abilities of an organization, creating a structured domain-dependent framework to define a resilience profile at different levels of abstraction, and identifying weaknesses and strengths of the system and potential actions to increase system's adaptive capacity. An illustrative example in an anesthesia department clarifies the outcomes of the approach.

Conclusion

The outcome of the resilience analysis grid, i.e., a weighed set of probing questions, can be used in different domains, as a support tool in a wider Safety-II oriented managerial action to bring safety management into the core business of the organization.

1. Introduction

Several changes are occurring in the business and socioeconomic environment, contributing to increase the complexity of work practices in sociotechnical systems. A sociotechnical system is a purposeful structure consisting of interrelated and interdependent social and technical elements influencing one another, directly or indirectly, to maintain their activity and the existence of the system itself to purse its goal [1], [2]. Sociotechnical systems are made up of people who produce products or services using some technology, affecting the operation and the appropriateness of the technology as well as the actions of the people operating it. Their interactions partly comprise linear cause–effect relationships and partly nonlinear, complex, even unpredictable ones. In a sociotechnical system, two sub-systems are tightly interrelated: the social and the technical systems. The social subsystem comprises of the people working within the organization at different levels (employees, managers, contractors, etc.) and the relationships among them (behaviors, activities, communication, etc.) [3], [4], even in terms of work attitudes, skill levels, and occupational roles. The technical subsystem consists of the artefacts used to convert inputs to outputs, including materials, machines, territory, and techniques used to accomplish the tasks of the organization itself [5], through work processes, roles, and procedures strictly related to with the social subsystem. Examples of sociotechnical systems are the air traffic management system, a production plant, maritime operations, and healthcare practices. For these systems, following the technological advances in the last decades, procedures and organizational activities have become increasingly more sophisticated, leading to increase system's complexity. Individuals have to thus cope with a challenging environment, whose complexity inherently emerges in everyday activities, as well as in design, implementation, and maintenance.

Traditional approaches in safety management focus on adverse outcomes (accidents, incidents) trying to reduce their numbers and/or limiting their effects. Following a linear approach, also labeled Newtonian reasoning [6], traditional safety management aims at reducing the variability of everyday work through stricter regulations and procedures to constrain work activities. This approach relies on the causality credo, i.e., the possibility to find a specific cause–effect link for any event, decomposing the system in its constituent parts, and generally thinking about humans as a source of failures, as summarized by the concept of Safety-I [7]. Nevertheless, Safety-I assumptions are effective only if system's complexity is limited. In complex operating scenarios, such as nowadays sociotechnical systems, there are many strong interactions among human, technical, organizational, procedural, and regulative aspects that lead to lose an effective linear understanding of the whole system functioning. The acknowledgment of the inevitable complex nature of sociotechnical systems pushed safety management research toward focusing on a different perspective, i.e., aiming at understanding everyday activities. In everyday activities, work-as-done is different from work-as-imagined, and this difference requires humans' adaptation to cope with variable operating scenarios and safely ensure system's productivity safely. This adaptation is embodied in the concept of resilience.

Resilience—at large—can be defined as a system property that confers the ability to remain intact and functional despite the presence of threats. Thus, resilience engineering is the ability to engineer systems that are resistant to disturbances. As proved by successful applications in several sociotechnical domains, mainly humans' resilience acquires a crucial role to maintain system's effectiveness and positive capabilities. Cognitive tasks (e.g., decision-making, inference, reasoning, learning, etc.) become central for maintaining safe and efficient operations.

There are several examples detailing the relevance of resilience in sociotechnical systems, for example, adaptive capacity of air traffic controllers and pilots has been modeled to understand how variability is managed to prevent runway incursions during normal runway operations [8]. Similarly, in the rail traffic domain, the crucial role of traffic controllers in coping with increasingly complex automation and ensuring system resilience has been acknowledged [9]. Resilience becomes necessary to deal with the synchronic and diachronic couplings of system functioning, as proved by a resilience engineering perspective used to analyze the Fukushima disaster [10]. Note that resilience is not by itself the solution for improving system performance, but by adopting a systemic approach it will be possible to understand where resilience is necessary and where it is not, as theoretically discussed in the healthcare domain [11].

For this purpose, one of the principal efforts in the discipline of resilience engineering consists of anchoring the concept of resilience to some clearly describable features to develop operational means for its assessment [12], [13].

Resilience engineering suggests four abilities to be considered for achieving resilient performance, i.e., responding, monitoring, learning, and anticipating. The resilience analysis grid (RAG) aims at measuring how resiliently an organization performs in everyday work based on assessing how the organization responds, monitors, learns and anticipates in everyday activities. Therefore, the RAG is developed as a questionnaire to produce a resilience profile of the organization, in terms of the four abilities of resilience, the so-called four cornerstones [14].

With the purpose of measuring organizational resilience in a complex system following a systematic and structured approach, this article aims at answering the following research question:

• -How should a method for measuring organizational resilience in a sociotechnical system be designed to consider the functional properties of the system and be used as a decision support tool to address criticalities, in a safety management perspective?

Starting from the traditional formulation of RAG, this article proposes an innovative analytic hierarchy process (AHP) framework to represent the effects of each resilience ability (responding, monitoring, learning, and anticipating) to organizational resilience, by user-friendly indicators that can be analyzed at different levels of abstraction. This framework allows dealing with the complexity of the system itself and identifies weakness and strengths of work activities, which remains a puzzling activity for managing system's resilience.

The remainder of the article is structured as follows. Section 2 describes materials and methods of this article, presenting the relevance of the four cornerstones of resilience and detailing their meaning in different work domains. Section 3 discusses the traditional RAG structure and its enhanced approach based on AHP, including an illustrative case study in an anesthesia department. The Section 4 offers critical reflections on the outcomes of the study, clarifying the potential for further research. Lastly, the Section 5 summarizes the outcomes of the study.

2. Materials and methods

Resilience is defined as the system's ability to retain or recover a rapidly stable condition, enabling pursuing its normal functioning during, and following, any expected or unexpected disruptions. Resilience reflects the ability of trading-off between safety and production objectives, managing conflicting goals' pressures [15], [16], [17].

2.1. Resilience in four cornerstones

In the field of resilience engineering, several research efforts aimed at linking resilience to some clearly describable features. One of the most diffused approaches led to decomposing resilience into four cornerstones, acknowledging that a resilient system needs to be conveniently balanced among them [14]. Following this view, the system must be able to respond to any type of events (addressing the actual), to monitor underway evolutions (addressing the critical), to anticipate future threats and opportunities (addressing the potential), and to learn from past failures and successes (addressing the factual). The validity of these four cornerstones has been widely recognized for representing successfully how people feel comfortable with unexpected and unforeseen events in everyday work [18] and to promote proactive strategies for managing daily operations [19]. The same cornerstones have been used to define a dedicated framework, which considers legal, institutional, organizational, and procurement aspects of societal resilience [20]. Even in the air traffic management context, the SCALES framework [21] and the ADAPTER questionnaire [22], both starting from the four cornerstones, have been used to retrieve resilience indicators and patterns to identify criticalities and the focus of more detailed approaches.

2.2. Responding: dealing with the actual

Responding to external and internal disturbances or, more in general, to any input or signal is a crucial need for every system. A system has to be able to distinguish between what is urgent and what is important and provide effective and on-time responses to maintain productivity and ensure safety. This distinction is necessary to ensure that the real-time competences that needed to cope with unanticipated or extreme events at the sharp-end are not eroded in the continuous attempt to predetermine corresponding responses to all the possible events [23]. Responding is linked to sharp-end decision-making: managing potential variability of patient status and defining the cognitive threshold over which requiring additional resources is generally a difficult task [24]. Several examples of prompt responding to system disruption can be retrieved in the health-care literature, especially in the emergency departments, which demands for strong responding capabilities due to the inevitable variability of everyday practices [25], [26], [27].

As a summary, the analysis of this cornerstone should aim at providing material for understanding what capacities are required to properly respond to everyday situations.

2.3. Monitoring: dealing with the critical

The ability of responding is linked to the capability of understanding actual threats timely and precisely. An effective responding has thus to be linked to an effective monitoring of system's status and of its operational environment, i.e., acquiring signals related to both positive and negative situations.

In traditional safety management [7], monitoring consists of adopting indicators such as the number of adverse events: mainly accidents [28] and also other less-consequence events, such as incidents and near misses [29], are used generally as feedback signals of system safety. On the contrary, for resilience engineering, it has been claimed the need to gather data from intermediate status, monitoring how the system actually performs during its normal functioning [30]. This target is generally acquired by leading indicators, i.e., precursors to events that are not happened yet, and/or lagging indicators, i.e., observations of events already happened. Note that a lagging indicator related to the short-term staffing change (e.g., numbers of workers suffering from fatigue-related injuries) might be considered a leading indicator for the need of systemic changes [31].

The analysis of this cornerstone should aim at defining relevant indicators to gain knowledge of current working conditions.

2.4. Anticipating: dealing with the potential

In a traditional sense, the ability of anticipating is related to forecasting future events based on historic data [7]. Even if these representations offer an overall understanding of safety levels [32], they might fail at anticipating hidden and new—never happened before—threats or even at determining the need for buffers and reserves to face unexpected pressures.

What makes anticipating different from monitoring is generally the different time scale of observations and the related point of view. The ability of anticipating generally extends the focus of monitoring's leading indicators to cope with long-term changes, threats, opportunities, and environmental potential status. At different organizational levels, monitoring and anticipating might become complementary. For sharp-end operators, coping with near-term issues is an operational everyday activity (monitoring); at the blunt-end, more attention is paid generally to strategic decision-making, relying on long-term analysis (anticipating). The distinction is not a dichotomy because strategic decisions affect and are affected by operational behaviors and vice versa [33]. Typical anticipating features are related to understand if, and how, future events like threats and opportunities are modeled (e.g., qualitatively, quantitatively, etc.), understanding which efforts in terms of expertise and funding are employed for this purpose [34].

The analysis of the ability to anticipate should look at detecting upcoming threats and opportunities timely and efficiently and increase system's preparedness.

2.5. Learning: dealing with the factual

Traditional safety management bases learning on adverse events, i.e., accidents, incidents, near misses. Normally, in safe systems, this assumption clashes with the principles of effective learning: the presence of reasonable opportunities to learn (a reasonable number of situations); the similarity of situations to avoid having only event-specific conclusions (the events should be comparable in some sense); the opportunity to verify whether the lessons have been learned [35]. Therefore, it becomes necessary to integrate the learning from accidents, or even more extreme events such as emergencies and catastrophes, with learning from minor-consequence events or even from normal functioning, namely everyday activities [14]. It is from lessons learned that an individual, or even an organization, increases his/her ability to manage threats, adopting or adapting (refer to Weick's sensemaking theory [36], [37]) his knowledge creatively to manage unexpected situations [38], [39]. At the same time, an effective learning should lead to choose worthy synthetic indicators and give means to anticipate potential future threats and opportunities.

2.6. RAG in literature

In accordance with the need to understand and monitor resilience [40], intended as a combination of the four cornerstones, Hollnagel originally developed a questionnaire-based tool in 2011, the RAG [41]. Hollnagel himself suggests that the RAG is not an off-the-shelf tool, but it is rather intended as a basis to develop questions, which must be specific for the system under analysis.

One of the first applications of RAG is discussed in the domain of rail traffic management, aimed at identifying a set of potential vectors for improvement, with particular reference to the need of integrating trade-offs and making easier to assess and control nuances of the abstract cornerstones [42].

For the purpose of integrated planning of maintenance and operations in an offshore oil and gas company, 16 critical functions have been assessed by a list of RAG-based questions. The study points out the benefits of adopting RAG after a system change to understand the effects of the change on the organization at least once a year [43]. A case study for the air traffic management system details the application of RAG to measure organizational resilience, in collaboration with several domain experts. The study confirms the benefits of repeating the RAG several times during a long period of time [44]. In the health-care domain, the RAG has been used to derive probes for each cornerstone, creating an interview script iteratively revised based on responses to pilot interviews as well as feedback by human factors and resilience engineering experts [45]. In addition, a recent study based on RAG in Polish enterprises of different sizes and activities shows how the implementation of occupational safety and health management system does not actually affect safety and resilience levels [46]. For the purpose of measuring team resilience, the RAG has been even updated by two additional dimensions, i.e., leadership and cooperation, suggesting its use for contextualizing actual events and having it filled after specific incidents [22]. This recent interest in the RAG formulation motivates the need to explore its building process to increase its potential benefits for sociotechnical systems.

2.7. RAG's building process

The RAG consists of four phases [41]:

• -RAG 1. Define and describe system's structure, boundaries, time horizon, people, and resources involved.

This preliminary phase is necessary to restrict the field of application of the analysis, contextualizing the set of questions to ask.

• -RAG 2. Select relevant questions for corresponding relevant items of the studied system.

This phase requires large efforts to define a proper set of questions which would be reliable enough to describe the system, not being so large to require unmanageable processing times. Even if there are several abstract commonalities across different high-risk domains of what the cornerstones mean [47], [48], it is hardly possible to define general questions valid for each domain. Since resilience is strongly related to the system's purpose for which it has to be assessed, the four cornerstones have to be detailed in a domain-specific context. This phase is usually a recursive activity to formulate the more appropriate questions, involving several subject–matter experts (SMEs).

• -RAG 3. Rate the questions for each cornerstone.

Once the questions are finalized, it is necessary to identify a pool of people working in the system who could answer them. In this sense, techniques to collect data are unconstrained; it is thus possible to develop informal phone or face-to face interviews [46], open questions in focus groups [42], narratives [45] or surveys. The surveys represent the most frequent technique [22], [43], [44] as they allow faster processing times, even if they do not allow dynamic interactions in data collection: for the interviewee to recall experience and for the interviewer to fill eventual gaps and inconsistencies in the judgments.

• -RAG 4. Combining the ratings.

In this phase, data are usually presented in a star plot where each axis corresponds to the variables used to rate each cornerstone. It might be possible to obtain an additional four spoke star chart combining the individual cornerstones' charts, to gain a description of the organizational resilience.

Note that the star plot is not a measure of resilience per se, but it represents in a compact way how the abilities are rated in a specific time moment: it represents a snapshot of organizational resilience under specific conditions. Therefore, it might be performed multiple times to follow and monitor performance developments. The RAG can be used firstly to determine where the system is; then to spot where the system should be; finally to understand how the system may reach a target status.

Nevertheless, the traditional outcomes of the RAG fail to represent the relative importance of each category and questions and their impact on the organizational resilience (the corner of a star plot do not offer any insights on these aspects). Although these open issues have been widely discussed according to a theoretical perspective acknowledging the need to understand the domain-specific balance among the abilities [41], there is no operational solution in literature. This research aims to fill this gap, through an innovative analytic structured formulation for the RAG based on the AHP.

2.8. The AHP for the RAG

The AHP, introduced by Saaty [49], is a multicriteria decision-making technique, which aims at reducing complex decisions to a series of pairwise comparisons, in a user-friendly formulation. In terms of the RAG, the AHP might be used to combine objectively subjective judgments and determine the relative weight of each variable, i.e., defining the effects on organizational resilience of each category and question [50]. To adopt the AHP, it is assumed that data collection will take advantage of a structured survey. In terms of the four traditional phases of the RAG, the AHP aims at structure of the second phase of the RAG, i.e., RAG 2, through a systematic framework to determine and weigh the final set of questions. The AHP's outcomes constitute an input data for the traditional third phase, RAG 3 and obviously inherently affect RAG 4.

The AHP steps, contextualized for the RAG application, can be described as follows:

• -AHP 1. Define the problem and determine the kind of knowledge sought.

The purpose of this phase consists of determining the factors that influence organizational resilience. This step thus corresponds to the preliminary activities performed for the second phase of RAG, i.e., RAG 2.

• -AHP 2. Structure the decision hierarchy.

This step allows defining the hierarchy of the problem, from the top with the goal of the process, through intermediate levels (criteria on which subsequent elements depend) to the lowest level (which usually is the set of alternatives but here represents the different safety events). In term of RAG, this phase consists of defining a hierarchy of resilience abilities: the overall organizational resilience lies at the top level, while the four cornerstones lie at the intermediate levels, and at the lowest levels, there are the so-called categories and, eventually, sub-categories. The lowest level shall include the final questions. The need to develop questions in a hierarchical structure affects the traditional second phase of RAG, implying a systematic reorganization of the RAG questions. Even if this task may generate additional efforts, it actually offers an inherent support in developing the questions themselves and guiding the analysts to follow a structured approach by the support of a multilevel reasoning [51]. Note that this step has to take into account the experts' judgments and consists generally of an iterative procedure (see Fig. 1 details of a real scenario, as discussed in the case study).

• -AHP 3. Elaborate a set of pairwise comparison matrices.

Fig. 1.

Process flow for phase 2 of RAG based on the AHP.

AHP, analytic hierarchy process; RAG, resilience analysis grid.

Each element in an upper level compares to the elements in the level immediately below. Pairwise comparisons express a relative judgment between two elements in a 9-degree scale of importance (1 = equal, 3 = moderate, 5 = strong, 7 = very strong, 9 = extreme) and the reciprocal value is assigned to the inverse comparison. In the elements of the pairwise comparison matrix A, a_ij represent the importance of i-th criterion over j-th one. Since the elements satisfy the constraint a_ij × a_ji = 1 and a_ii = 1, only n (n − 1)/2 comparisons are necessary. At this step, the consistency index I_C has to be evaluated based on the eigenvalues λ of the matrix A [49].

If I_C < 0.1, the matrix is consistent, and the judgments can be considered not contradictory. For the purpose of applying the RAG, the comparison at the intermediate level, i.e., the four cornerstones, might be too abstract for evaluation and thus their weights might be considered equals. Lower levels of abstractions might be instead evaluated following traditional AHP theory.

• -AHP 4. Use the priorities coming from the comparisons to weigh the priorities in the level immediately below.

This weighing process has to be repeated for each element, also including the weight of the upper category to gain an overall indication of their relative importance. The hierarchy composition principle has to be applied to determine the importance of every element for organizational resilience, which represents the main goal in the AHP language. Therefore, starting from the lowest level, the local weight of every element has to be multiplied for local weight of the related parent element, transforming them in global weights.

In terms of the RAG, once the AHP is applied and the elements' weights are identified —both local and global—at each level of the hierarchy, it is possible to continue with the traditional survey, as discussed in the third phase of RAG, i.e., RAG 3. The AHP's structure will inherently modify and enhance the data management in the fourth phase of RAG, allowing detailed comparisons among questions and their effects on the system, as clarified by the illustrative example in Section 3.

3. Results

This section details an illustrative application in the health-care domain of the innovative RAG analytic framework based on the AHP. Mainly due to the instrumental, procedural, and organizational continuous evolution, new challenges constantly affected health-care system. The human interaction with these factors became steadily more complex, requiring resilience as a means to manage the variability of everyday performance [52], [53]. Understanding how resilient is a health-care organization might represent a valuable contribution to determine how effectively the organization manages the complexity of everyday work, defining weaknesses and strengths.

3.1. RAG phase 1

This illustrative application focuses on an anesthesia department of a hospital located in Central Europe. The aim of the study consists of assessing the organizational resilience of neuroanesthetists working in the same department, by a RAG-based weighted survey. The target of the study consists of assessing how the individuals working in the department contribute to organizational resilience in terms of the four cornerstones, appropriately contextualized for the need of a neuroanesthesia domain.

3.2. RAG phase 2

The AHP guides the development of this phase. Starting from a literature review on resilience engineering, specifically focused on previous applications of the RAG, an initial set of categories (Level 2 of hierarchy) and questions (Level 3) has been created. In the hierarchy, formally the four cornerstones occupy Level 1, while the overall organizational resilience represents Level 0.

A focus group (one staff and two resident neuroanesthetists and two researchers working in the field of resilience engineering) tuned this initial framework to adapt it properly for the needs of the specific neuroanesthesia department.

After this phase, a preliminary version of the survey has been tested with four physicians: two staff and two resident neuroanesthetists. This test led to modify the framework slightly, reducing the size of questions and rephrasing some of them to make them clearer and avoid misunderstanding.

Fig. 1 summarizes the process and Table 1 details the final list of categories (Level 2 of the hierarchy). Note that the staff doctors answering the preliminary survey were not the same involved in the focus group. Theoretically, it would have been necessary to develop pairwise comparisons at each level of the framework. However, for operational reasons, the AHP weighing process has been restricted to the Level 2 of the framework. The comparisons at Level 1 would have been too abstract to obtain meaningful and unambiguous responses; and the comparison at Level 3 would have been of limited added value, considering the reduced size, i.e., two or three questions for each Level 2 category (also considering the extra resources necessary in this additional weighing process). Consequently, the weights of Level 1 and Level 2 have been thus assigned automatically as equally important, respectively 0,25 for each cornerstone, and 0,50 or 0,33 for each questions, in case the category included respectively two or three questions.

Table 1.

List of RAG categories for each cornerstone (AHP Level 2)

Responding	Monitoring	Anticipating	Learning
R1. International guidelines	M1. List of questions for preoperative examination	A1. Improvement projects	L1. Discussion of clinical cases
R2. Internal protocols	M2. Preoperative specialist support	A2. Research activities	L2. The register of clinical cases
R3. Resources availability in expected situations	M3. Intraoperative risk analysis	A3. Introduction of new equipment	L3. Event analysis
R4. Resources availability in unexpected situations	M4. Identification of intraoperative complications	A4. Participation in updating plans and procedures	L4. Practical activities
R5. Experience	M5. Instrumental support	A5. Planning of training activities	L5. Theoretical teaching
R6. Discretionary power	M6. Postoperative check	A6. Surgeries scheduling	L6. Mentoring
R7. Teamwork	—	A7. Communication	L7. Learning check
R8. Roles division	—	—	L8. Experiences in other institutes

AHP, analytic hierarchy process; RAG, resilience analysis grid.

Afterward, based on the final structure of Level 2 (see Table 1), seven neuroanesthetists (the ones involved in the focus group and in the preliminary test) help defining the weights for the categories, individually filling an online survey. This allows pairwise comparison among the categories, based on a 9-point scale (A and B equally important, A slightly more important than B, A mildly more important than B, A much more important than B, A absolutely more important than B, and vice versa). It has been chosen to ask for individual answers, rather than collective ones (e.g., in a focus group) to avoid that individual attitude to team working, or organizational role might have influenced judgments. Then, a grouping strategy for the judgments has been applied relying on geometric mean [54]. These surveys were not anonymous, in order to ease the verification of judgments' coherence following an iterative approach to verify I_C < 0.1 for each judgment, as described in [54]. The outcome of this phase represented the final set of weighed questions, as summarized in Fig. 2 and more specifically in Fig. 3.

Fig. 2.

AHP structure and weights for each cornerstone, category, and question.

AHP, analytic hierarchy process; RAG, resilience analysis grid.

Fig. 3.

Details of nomenclature and example of questions.

AHP, analytic hierarchy process.

3.3. RAG phase 3

At this step, it has been necessary to rate each element at Level 3 of the hierarchy, i.e., the probing questions, by a specific survey. The survey was anonymous, gathering few profile data on the interviewee (sex, age, role in the hospital, and years of experience). The questionnaire included a description of the meaning of each category and a 5-point Likert scale, linguistically adapted for each question, as shown in Fig. 3. The survey has been submitted to 12 neuroanesthetists (six staff and six resident) of different ages and experience, working in the same department.

3.4. RAG phase 4

This phase summarizes the answers gathered from the 12 neuroanesthetists involved in the third phase of RAG, suggesting an interpretation of the results. As discussed in literature, one of the main advantages of using the RAG consists of monitoring the relative evolution of organizational resilience (e.g.,) repeating the assessment over time [45]. However, for this illustrative application, a single preliminary assessment will be discussed. Nevertheless, other potential ways to interpret RAG data are presented, taking advantage of the AHP structure. At Level 1 of the hierarchy, it is possible to represent how the organization performs respectively for each cornerstone (see Fig. 4). As a preliminary observation, one can note how the system appears to be much more able to respond and monitor, rather than to anticipate and learn. This abstract conclusion, obtained by combining all the judgments at each level, can be discussed further in terms of the categories for each cornerstone, as detailed in Fig. 5. This analysis allows detailed observations, detecting which categories mainly contribute to overall resilience. For example, in terms of learning, anesthetists feel quite comfortable about event analysis, i.e., reactive and proactive investigations undertaken following an accident or a risky event to detect contributing factors. About the ability of responding, experience has a crucial role, as expected. In addition, the study shows that resources availability cannot be considered the most critical aspect, regardless if considering expected or unexpected events.

Fig. 4.

The RAG score for the four cornerstones.

RAG, resilience analysis grid.

Fig. 5.

The RAG score for each category at Level 2 of the hierarchy.

RAG, resilience analysis grid.

About Monitoring, even if real-time observations of complications during intraoperative phase seem to be effective, the analysis shows some drawbacks about the preoperative specialist support. This conclusion considers those frequent scenarios where the anesthetist needs support to further investigate the patient's clinical conditions, but he/she does not receive a proper support, mainly due to delays and problems for interacting with other departments.

In terms of Anticipating, the interviewees show a strong confidence in communication with surgeons and nurses. Communication's high score shows how the verbal interactions among different practitioners allows anticipating future threats and mitigating emerging risks in patient safety, especially in case of shift rotation over the same intervention.

Furthermore, additional observations can be drawn from the results of the RAG, through a cluster analysis. Thinking of resilience as a multilevel property of a system, the purpose of this cluster analysis consists of identifying how the individuals contribute to organizational resilience, assuming that the organization provides the context for individuals or group of them to make local and global system properties emerge [55]. On this path, even if the sample is limited, it allows interesting observations about how resident and staff neuroanesthetists differently contribute to the overall organizational resilience. This cluster analysis is intended to tailor the definition of criticalities and the development of potential mitigating actions to manage system's resilience.

More specifically, at Level 1, one can observe that resident neuroanesthetists offer a lower contribution to organizational resilience (see Fig. 6), intuitively related to their experience in the field. This abstract observation can be broken down taking advantage of the AHP hierarchy to gain more detailed and helpful insights.

Fig. 6.

The RAG score for each category at Level 1 of the hierarchy and comparison between the resilience profiles of staff and resident neuroanesthetists.

RAG, resilience analysis grid.

Fig. 7 offers a two-dimensional representation to give an overview of the categories where the staff neuroanesthetists' contribution to overall resilience prevails on the residents' one (the area under the dashed line), and vice versa (the area above the dashed line). This analysis shows that in some specific categories the resident neuroanesthetists are more resilient than the staff ones, i.e., L.4 (practical teaching activities); R.1 (international guidelines); M.3 (intraoperative risk analysis), M.6 (postoperative check); A.2 (research activities), A.6 (surgeries scheduling).

Fig. 7.

The RAG score for each category at Level 2 of the hierarchy and comparison between the resilience profiles of staff and resident neuroanesthetists.

RAG, resilience analysis grid.

A focus group with three doctors (the same involved in the focus group during phase two of RAG, see Section 3.2) allowed drawing interesting conclusions. For example, about Research activities (A.2), the focus group confirms that resident neuroanesthetists generally play a major sharp-end role in trials and experimental projects. These activities force them to keep abreast of anesthesiology scientific evolution, increasing their anticipating skills, i.e., their resilience. This factor has been perceived as positive for the benefit of the organization, indicating the need at a managerial level to promote an active involvement of staff anesthetists in research activities.

On the contrary, about the aspect Participation in updating plans and procedures (A.4), staff neuroanesthetists prevail on resident ones, since the latter are not usually involved in the planning processes. Staff doctors, whose expert opinion is generally predominant, feel more comfortable to manage procedures and plans in which they have been directly involved. The results of the RAG motivate a managerial action to encourage the involvement of people with fresh eyes, i.e., resident doctors, in medium and long-term planning. The proposal of these two managerial actions aim at fostering multiskilled anesthetists with interchangeable roles and distributed workloads, in line with 1950s seminal observations in mining systems [1], and as recently confirmed in the health-care domain (see [56] for an example in operating room and [57] for information technology management). At Level 3, further observations can be drawn, based on scores of specific probing questions. For example, local improvement actions might consist of increasing the number of teaching activities with mannequins (L.4.1) and stressing the relevance of using a complete list of questions during the preoperative visit (M.1.1). For this latter aspect, the results of the RAG motivated a detailed investigation of everyday work, which confirmed that the preoperative visit is frequently underrated for apparently healthy patients, generating potential complications during the intraoperative phase for underestimated risk factors. At management level, the development of a dedicated training action to underline the need of an accurate and detailed preoperative visit, regardless of the patients' conditions, has been strongly encouraged.

4. Discussion

Measuring resilience is a challenging task, mainly because it requires an empirical, context-dependent measurement tool. On this path, since resilience refers to something that a system does, rather than something it has, the RAG in combination with the AHP defines a resilience profile that may offer insights to understand when and where it is possible to intervene to manage work activities in a way capable of sustaining the required operations.

When assessing an inherent property of a system, e.g. its resilience, it is important to avoid making important what is measured, rather than measuring what is important, defining relevant proxy measures as for the proposed approach. Therefore, it is useful to remember that the RAG profile, even in combination with the AHP, does not offer an absolute rating of system's abilities but a relative assessment support tool to compare different RAG profiles. The RAG can be used to assess how the resilience profile of an organization changes over time, using it repeatedly with the same group of respondents and/or to assess resilience profiles for different groups of respondents (as discussed in the case study).

As emerged in the case study, there are no predetermined set of indicators, but they have to be specific to the context and the risks that exist in a specific scenario. For this purpose, the involvement of SMEs working within the organization (with a job related to the target of the assessment) acquires a crucial role, as understandable through the lessons learned from the case study presented in the article. It is important to gain a strong consensus among SMEs in the process of defining the final set of questions, since this will clearly affect the definition of the resilience profile and the identification of the mitigating actions. Sharp-end operators have to be involved among the SMEs in the development of the RAG, in order to get a meaningful and reliable set of questions which can define a resilience profile that is able to describe how the system cope with the complexity of everyday activities. While developing the RAG in combination with the AHP, the analyst has to limit the efforts of the respondents, trying to define a reasonable number of questions. As emerged from the case study presented in this article, the most time-consuming phase is the weighing process of each category through specific pairwise comparison matrices. For this purpose, we developed an online user-friendly survey taking advantage of Google Survey, which allowed respondents to answer questions directly from their computer, tablet, or smartphones, limiting the perceived efforts in the task. The format of the tool was extremely appreciated by the respondents, who were willing to contribute even in the iterative procedure to maintain the coherence of the pairwise comparison judgments, as explained in Section 3.2. Similarly, another survey to rate each probing question has been developed in Google Survey, as needed for the third phase of RAG.

The framework and the structure of the online surveys can be adopted in different departments or organizations, but requiring the contextualization of the domain-dependent categories and variables.

Even if the AHP does not limit the number of hierarchical levels, lessons learned from the case studies suggest that a three-level structure adequately represent the need of a sociotechnical assessment. More specifically, Level 2 is a reasonable level of abstraction to weight the factors, with the possibility to avoid too much abstract weighting process (at Level 1, for the four cornerstones), or too detailed ones (at Level 3, for the questions). Also note that the AHP could be used to rank factors and exclude from the analysis those categories that receive substantially lower weights than others, under the assumption that their effect on organizational resilience is limited. This assumption will eventually require a renormalization of the other weights, generating a potential reduction of the number of questions to be answered.

In line with the traditional RAG approach, multiple representations may be helpful to describe the relative status of the system or the effects of a systemic change in the resilience profile. In this context, a limitation of the illustrative case study presented in the article is the limited size of the sample, which has to be increased in future research. Furthermore, a larger sample would have allowed multiple cluster analyses to compare different groups of respondents and gain multiperspective insights. Similarly, although the illustrative case study presents a single assessment of the resilience profile, capable of generating valuable insights, a systemic analysis would largely benefit of multiple assessments over time to assess the evolution of the resilience profile of the organization (as a whole or for different clusters).

In terms of future research, for potential adaptation of the framework presented in this article to different domains, it is helpful to recall that the analyst might even not limit the RAG to the four cornerstones, extending or modifying the categories, if other abilities are specific to a work domain.

Future research might even explore the possibility to extend the analysis by a network structure, based on the analytic network process, allowing assessment of interaction and dependence of higher level elements in a hierarchy on lower elements. However, in this case, the analytic structure will be more complicated, requiring more time-consuming pairwise comparisons. In terms of methodological formulation, other approaches to gather data could be evaluated as well, i.e., interviewing people to get open answer and ensure the interviewees completely understand the questions. Even in this case, more resources will be requested for gathering data, in addition to the need of experienced interviewers.

Lastly, future research may focus on the development of a fuzzy AHP, to deal with the problem arising when making static and deterministic quantitative assessment. Owing to the linguistic values and variables necessary for the application of the RAG, the decision of taking the fuzziness into account might provide potentially less biased and more realistic assessments.

5. Conclusions

Within the scope of resilience engineering for safety management, this article suggests an approach to measure resilience by means of the RAG, a questionnaire-based tool based on the four cornerstones of resilience. Since resilience is a property of the system and is strictly dependent on the features of the system itself, the theory of resilience engineering does not prescribe a standard meaning of the four cornerstones or even a certain desired balance or proportion among them. Adopting the AHP, this article develops a structured approach to determine an in-depth meaning of resilience abilities, creating a hierarchy based on different level of abstractions. The RAG's survey data, systematically weighed following the AHP, represent a resilience profile that can be used to assess the system status and to address—if necessary—priorities for intervention (both for safety and performance enhancement), as confirmed by the outcomes of the illustrative case study presented in this article. It is important to observe that due to its inherent questionnaire-based nature, the RAG relies on truthful answers and, consequently, on an interviewee who wants to exaggerate in one sense or the other his/her work conditions, which can seriously affect its value. For this purpose, even if increasing the number of participants may reduce considerably the risk of facing this bias, there is the need to develop the RAG as part of a wider Safety-II oriented managerial action with focus on sharp-end activities. In this local analysis, the outcome of RAG could even motivate detailed work-as-done research, using methods to analyze systemic interactions and functional properties, e.g., through the Functional Resonance Analysis Method (FRAM) [58] and/or the Systems-Theoretic Accident Modeling and Processes/Systems Theoretic Process Analysis (STAMP/STPA) [59]. On this path, the RAG may represent a tool to bring safety management into the core business of the organization to support the development of a healthy and efficient work environment.

Conflicts of interest

All authors have no conflict of interest to declare.