Fatigue Risk Management Systems Diagnostic Tool: Validation of an Organizational Assessment Tool for Shift Work Organizations

Abstract

Background

This study aimed to determine and define the elements of an Fatigue Risk Management System (FRMS) diagnostic tool to assist an organization in systematically assessing its level of implementation of an FRMS.

Methods

A modified Delphi process was used involving 16 participants with expertise in sleep science, chronobiology, and fatigue risk management within occupational settings. The study was undertaken in two stages 1) review of elements and definitions; 2) review of statements for each element. Each stage involved an iterative process, and a consensus rule of ≥ 60% was applied to arrive at a final list of elements, definitions, and statements.

Results

Stage 1: a review of elements (n = 12) and definitions resulted in a final list of 14 elements and definitions with a consensus of ≥ 60% achieved after 2 Delphi rounds. Stage 2: a review of statements (n = 131) resulted in a final list of 119 statements with a consensus of ≥ 60% achieved after 2 Delphi rounds.

Conclusion

The final FRMS diagnostic tool will enable an organization to systematically assess the level of implementation of their current FRMS and identify gaps and opportunities to reduce risk.

1. Introduction

Fatigue results from sleep loss that may adversely impact an individual's alertness and increase their accident risk due to an increase in their time to react (reaction time), reduced ability to make decisions, and poor concentration [1,2]. Furthermore, fatigue is associated with an increased risk of developing chronic health conditions such as obesity, diabetes, cancer, cardiovascular disease, and mental health disorders [3]. Shift workers are such an ‘at-risk’ group as they are required to modify their sleep-wake patterns to align with their roster schedule and work long hours [4], which may result in acute sleep loss across a roster schedule (< 7 hours sleep per day/night) [5]. This may result in what is commonly referred to as fatigue, defined as the “result of sleep loss (<7 hours) and/or being awake during an adverse circadian phase, thereby potentially affecting alertness” [6]. The injury rates for shift workers have been reported as over double that of non-shift workers [7]. Therefore, the effective management of fatigue risk is critical to protecting the health and safety of shift workers and ensuring organizational success [8], specifically for high-risk shift work organizations such as mining, oil and gas, aviation, transportation, and health care.

In addition to an increased risk of accidents and chronic health conditions fatigue has been associated with increased rates of absenteeism and presenteeism. It is estimated that sleeping < 6 hours per night may result in 6 working days lost per year due to absenteeism and presenteeism compared to those who achieve the recommended > 7 hours sleep, an average productivity loss of 2.4% [8]. The combined economic cost of sleep loss is significant with estimates of $66.3 billion, due to loss of well-being (60%), productivity losses (27%), other financial costs (9%), health system costs (3%) and informal care costs (1%) reported in Australia [9]. Therefore, improving the sleep of shift workers has the potential to not only improve health and safety outcomes but improve productivity and reduce costs to shift work operations.

In managing fatigue risk, shift work organizations frequently adopt a compliance-driven approach within their broader Safety Management System (SMS), that provides specific guidance and prescriptive limits on work hours based on legislation [4]. This approach in isolation may be insufficient to manage fatigue risk due to the multifaceted nature of fatigue [4,10]. For example, fatigue may still be experienced by workers despite compliance with legislation due to the presence of other factors such as sleep disorders [11]. It is increasingly recognized that a comprehensive systematic approach, referred to as a Fatigue Risk Management System (FRMS) may be more beneficial [4,11,12]. A FRMS allows an organization to operate outside of the set prescriptive work limits, by adopting an approach that is risk-based and data- driven, allowing increased flexibility and ensuring the unique risks of an organization are adequately addressed [13]. The core elements of an organization's SMS, that include policy, risk management, reporting, incident investigation, training and auditing, are also an essential part of an FRMS [4]. Depending on the maturity of an organization's SMS they may choose to manage fatigue risk as one of the hazards as part of their SMS or develop a FRMS to specifically manage fatigue risk [13].

For an FRMS to be effective an organization should ensure that controls are implemented at each of the levels that commonly precede a fatigue-related incident. For example, such controls should ensure (i) adequate sleep opportunity is provided; (ii) actual sleep is obtained; (iii) the detection and monitoring of behavioral symptoms of fatigue and fatigue-related errors; and (iv) incident analysis and investigation procedures are implemented following an incident [10].

Throughout the scientific literature, the elements required as part of a complete FRMS are not clear and have varied between studies [11]. Elements have included education and training, sleep disorder screening, actigraphy, cognitive tests, self-reported fatigue, and roster design assessment using biomathematical modeling [4,11]. In the transportation industry, thirteen elements that should be considered in monitoring and controlling fatigue risk have been identified [14]. A further review identified risk factors and outcomes of fatigue that will inform the development of a national fatigue risk management standard for first responders in Canada, including police, firefighters, and emergency medical services personnel [15]. However, no studies were found on the practical implementation of a comprehensive FRMS, including the use or validation of a tool to assist organizations in systematically assessing their current level of implementation of an FRMS against best practice recommendations. Despite this, high-risk industries such as aviation and oil and gas have developed guidance documents that include recommendations to support the development, implementation, and monitoring of an FRMS [13,16].

To support the practical implementation of an FRMS the research team had previously developed an FRMS diagnostic tool based on their industry experience and available literature [4,10]. This tool has been used across industries including mining, oil and gas and aviation with potential benefits to further assist organizations to identify improvement opportunities to reduce fatigue risk. Therefore, using the research teams already developed FRMS diagnostic tool as a base, this study aimed to (i) validate the elements of an FRMS diagnostic tool and (ii) develop an FRMS diagnostic tool to assist an organization in systematically assessing its level of implementation of an FRMS.

2. Methodology

2.1. Study overview

In this study, we utilized a modified Delphi process that included participants from a range of industries with expertise in fatigue risk management within industrial settings. This process examined the validity of the identified elements, definitions, and statements of an applied FRMS diagnostic tool. Participants engaged with the research team for two stages of the review, allowing up to three Delphi rounds. In addition, each stage required a level of consensus (≥60%) before advancing to the next. The Edith Cowan University Human Research Ethics Committee approved the study protocol (approval number: 2019-00813-MAISEY). Fig. 1 displays the modified Delphi process study design.

Fig. 1.

Modified Delphi process study design.

2.2. Study participants

To ensure diversity in the knowledge and expertise of the expert panel, the research team identified disciplines required to inform the development of the FRMS diagnostic tool, that included, sleep science, chronobiology, occupational medicine, legislation and regulation, occupational health and safety, and fatigue risk management in occupational settings. Based on these disciplines, an international expert panel was established, comprised of 16 participants who were identified based on their experience and contribution to these disciplines. Nine of the participants were selected based on their experience and knowledge specifically in fatigue risk management (Supplementary Table 1). Identified participants were formally invited to participate by email, at which time they were provided with a participant information sheet explaining the research project, to which they gave their consent. Participants were anonymized throughout the study. Only the lead researcher knew the participants' identification.

2.3. Fatigue risk management system diagnostic tool, elements, and statements

The research team had previously developed a systematic FRMS diagnostic tool to assist organizations to self-assess their current level of implementation of an FRMS. The initial FRMS diagnostic tool consisted of 12 elements (Table 1) with 131 statements based on the current sleep science, chronobiology, fatigue risk management literature, and the researchers' applied experience in industrial settings. The statements focused on evaluating an organization's policies, interventions, and programs that when implemented as part of a comprehensive system may reduce the risk of fatigue at an organizational level. The FRMS diagnostic tool is designed to be completed in its entirety, requiring the organizational representative to select one of the following responses to each statement:

• No - The organization does not meet the requirement of the statement.

• In progress - The organization partially meets the requirement of the statement.

• Yes - The organization meets the requirement of the statement.

• Not applicable - The statement is not relevant to the organization.

Table 1.

Elements and definitions

Element name and definition
1. Policy and Plan This construct assesses the fatigue management policy and plans for the business that provides guidance and identifies how the business will reduce the risk of incident, injury or adverse long-term health effects due to fatigue.
2. Leadership and management This construct assesses how leaders and managers engage and behave in a way that demonstrates their commitment to reducing fatigue risk, and influences workers to monitor and manage their fatigue.
3. Employee engagement This construct assesses how workers are engaged in the identification and management of fatigue within the business, including the development of fatigue policy, plans and programs.
4. Shift and roster design This construct assesses how the business designs and assesses shifts and rosters to determine the level of fatigue risk, and the level of compliance with government regulations.
5. Data and incident management This construct assesses how the business collects and analyse objective data to inform decisions regarding fatigue management, and if fatigue is considered as a causal factor in incident investigations.
6. Education This construct assesses if fatigue and sleep hygiene education is delivered to workers, their family and the broader community in which the business operates, as well as the quality of the education.
7. Healthy Lifestyles This construct assesses how the business supports and promotes lifestyle factors such as a healthy weight and the safe consumption of alcohol that promote good sleep.
8. Mental Health This construct assesses how the business supports and promotes a mentally healthy workplace, identifying fatigue as a psychosocial hazard for poor mental health.
9. Travel, Jet Lag and Commuting This construct assesses policies, plans and interventions that aim to minimise the risk of fatigue when commuting to and from work and whilst traveling interstate and overseas.
10. Sleep disorder management This construct assesses how the business identifies and manages workers with sleep disorders such as sleep apnea, insomnia and shiftwork disorder that may be exacerbating sleep loss.
11. Work and sleep environments This construct assesses the implementation of controls within the workplace to reduce fatigue and enhance alertness during work hours; and the design of sleeping environments where accommodation away from home is provided.
12. Fatigue technology This construct assesses how the business assesses and utilises fatigue technologies to detect and monitor worker fatigue e.g. In-cab, wrist-worn technology.

The target audience for this tool was Operational, Health and Safety, and Human Resources representatives of an organization with knowledge of the current state of their organization's FRMS, policies, and programs.

We used a modified Delphi process to establish the validity of this tool using the consensus rule of ≥ 60%, whereby participants were invited to provide their expert opinions on the elements, their definitions, and a series of statements to evaluate each element [17]. This technique has been used in many industries, such as health care, aviation, and mining [[17], [18], [19]], as well as in the health and safety field to develop and validate workplace assessment tools [20]. Our study was undertaken in two stages.

2.3.1. Stage 1: review of elements and definitions

This stage aimed to reach a consensus (≥ 60%) on the elements and definitions. Therefore, a maximum of three Delphi rounds were permissible. Participants were allowed up to 14 days to complete each round, with an estimated time commitment of 30 minutes. This stage was conducted using an online survey tool (Qualtrics™), and participants were asked to ‘’“agree” or ‘’“disagree” with the name and definition of each element, specifically (i) was the construct an integral element of an FRMS; (ii) was it evidence-based; and (iii) was the language used clear and did it accurately reflect the intent of the element. In addition, participants were asked to provide comments and suggest improvements based on their responses. Participants were invited to suggest additional elements and definitions with supporting justification for inclusion. After each Delphi round, responses were reviewed by the research team and considered for inclusion in the tool for the next round.

2.3.2. Stage 2: review of statements

Once consensus on the element names and definitions was achieved the next stage aimed to reach a consensus (≥ 60%) on the statements required to evaluate each element. Therefore, a maximum of three Delphi rounds were permissible. Participants were allowed up to 14 days to complete each round, with an estimated time commitment of 60 minutes. This stage was conducted using Microsoft Excel® and participants were asked to ‘’“agree” or ‘’“disagree” with each statement and provide comments and suggest improvements and any additional statements with supporting justification for inclusion. After each round, responses were reviewed by the research team and considered for inclusion in the tool for the next round.

2.4. Data analysis

Data were compiled using Microsoft Excel®. The percentage agreement for each element, definition, and statement was calculated to determine consensus. In addition, qualitative comments were analyzed to determine overall opinion or industry-specific nuances. The consensus rule of ≥ 60% was chosen for agreement with elements, definitions, and statements based on previous health science studies that reported consensus at ≥ 60% agreement [17]; and ≥ 60% was considered appropriate given the small number of participants of our study (n = 16).

3. Results

All invited participants agreed to participate in this study (n = 16). Combined, participants had experience across 15 different industries with 7–20+ years of experience and formal qualifications at bachelor's degree, Master's degree, and Ph.D. levels (Supplementary Table 1). The most common industry experience was mining (69%), oil and gas (56%), and government (50%). Participants' areas of expertise were occupational health and safety (69%), fatigue risk management (56%), and operations (50%) (Fig. 2).

Fig. 2.

Summary of participants' industry experience and areas of expertise. Notes: Data is presented as percentages (%). Participants were able to select more than one category for industry experience and area of expertise.

3.1. Stage 1: review of elements and definitions

Delphi Round 1: All participants (n = 16) completed the review. Overall, there was a high level of consensus for all 12 elements and definitions achieved an average consensus of 94% (81% - 100%) (Table 1). Five of the elements achieved 100% consensus: Shift and roster design; Data and incident management; Education; Travel, jetlag, and commuting; and Work and sleep environments. Participants provided comments and improvements on all elements despite consensus being achieved and suggested additional elements and definitions. The research team reviewed all comments, improvements, and additional elements and considered them for inclusion in the tool. As a result, the number of elements increased from 12 to 14 and were considered for review in Round 2 by the participants. The following elements were subject to change (i) Work and sleep environments were divided into two elements, Workplace hazards, and Sleep environment; and (ii) Fatigue risk management system implementation was included as an additional element.

Delphi Round 2: Ninety-four percent of participants (n = 15) completed this round of review. The 14 elements and definitions achieved an average consensus of 82% (67% - 93%). Participants provided minor comments and improvements, but no additional elements and definitions were suggested. The research team reviewed all comments and made minor improvements to the tool. Given that consensus ≥ 60% was achieved for all elements and definitions the study advanced to stage 2 of the process (Table 2).

Table 2.

Final elements, definitions and percent consensus

Element name and definition	% Consensus
1. Fatigue Risk Management System, including its policy, operational procedures and implementation plan This construct verifies the current state of the Fatigue Risk Management System, including its policy, operational procedures and implementation plan, and compliance with local legislation. This incorporates operational factors for the business and identifies how the business manages the risk of incident, injury or adverse health effects due to fatigue.	87
2. Leadership and commitment This construct verifies if leaders and frontline managers engage and behave in a way that demonstrates their commitment to reducing fatigue risk, and influences individuals to monitor and manage their fatigue.	93
3. Fatigue Risk Management System consultation This construct verifies if the business engages individuals in the management of fatigue, including the development of a Fatigue Risk Management System, and its policy, operational procedures and implementation plan.	87
4. Shift and roster design This construct verifies if the business designs shifts and roster patterns to identify and manage the level of fatigue risk exposure.	86
5. Health and safety data collection and analysis This construct verifies if the business collects and analyzese data e.g. from system reviews, audits, hazard identification and investigations to continuously improve the Fatigue Risk Management System.	73
6. Fatigue risk management and sleep science training and information This construct verifies if quality fatigue risk management and sleep science training and information are provided to individuals, their families and the broader community.	80
7. Healthy lifestyles This construct verifies if the business supports and promotes lifestyle factors that facilitate good sleep and reduce fatigue, such as the maintenance of a healthy weight and the safe consumption and use of alcohol and other drugs.	67
8. Psychological health and safety at work This construct verifies the identification and management of fatigue as a psychosocial risk and the promotion of mental well-being at work.	80
9. Travel and commuting This construct verifies if policies, plans and risk controls aim to minimise the risk of fatigue when traveling to and from work, including interstate and overseas travel that may result in travel fatigue and/or jetlag.	93
10. Sleep disorder management This construct verifies if the business identifies, manages and aids individuals with sleep disorders including sleep apnea, insomnia and shift work disorder that may be exacerbating sleep loss.	80
11. Workplace hazards and conditions This construct verifies the implementation of controls for workplace hazards and conditions to manage fatigue.	73
12. Sleep environment This construct verifies if the design of sleeping environments promotes adequate quality sleep.	93
13. Individual health monitoring and surveillance This construct verifies if the business effectively monitors the fatigue of individuals using health surveillance techniques.	87
14. Fatigue Risk Management System implementation and improvement This construct verifies if the business provides an effective process to implement and improve the Fatigue Risk Management System that considers its impact to all parts of the business.	73

[bold] was used to distinguish between the element (in bold) and it's definition.

3.2. Stage 2: review of statements

Delphi Round 1: Eighty-eight percent of participants (n = 14) completed the review. Two participants did not complete the review within the time allowed due to competing priorities. Consensus for all statements (n = 131) ranged from 57% to 100%. The mean percentage consensus for a group of statements assessing each element was ≥ 78%. Participants provided comments and improvements with suggested additional statements. The research team reviewed all comments, improvements, and additional statements to ensure they were evidence-based and not opinions before being considered for inclusion in the tool. As a result, improvements were made, and the total number of statements was reduced to 126 that were included for Round 2. This was due to duplication of statements within and across different elements.

Delphi Round 2: Eighty-eight percent of participants (n = 14) completed the review. The mean percentage consensus for a group of statements assessing each element was ≥ 81% (range: 71%–100%). In addition, participants provided minor comments and improvements. All comments and improvements were reviewed by the research team and as a result, improvements were made, reducing the total number of statements to 119 included in the final FRMS diagnostic tool. This was due to duplicating statements within and across different elements (Supplementary Table 2).

4. Discussion

This study reports on the modified Delphi process that was undertaken to (i) validate the elements of an FRMS diagnostic tool and (ii) develop an FRMS diagnostic tool to assist an organization in systematically assessing its level of implementation of an FRMS. The final FRMS diagnostic tool is presented as 14 elements, each element containing between 4 and 21 statements that assess the current level of implementation of that element.

The initial development of our FRMS diagnostic tool was informed by elements cited throughout the literature, including education, sleep disorder management, shift and roster design, and fatigue technology [4,10,11], as well as the International standards for Occupational Health and Safety Management Systems (ISO 45001) that provides guidance and a framework for minimizing health and safety risk in the workplace [21]. The research team's experience in applied FRMS and improvement allowed the initial FRMS diagnostic tool to be well developed with the correct elements primarily identified prior to the Delphi review process commencing. Therefore, we achieved a high consensus amongst the participants at the conclusion of Stage 1, Round 1. This enabled further development and refinement of the tool including the addition of elements and statements. We found a slight decrease in average consensus for the elements and definitions from round 1 (94%) compared to round 2 (82%), however, based on the comments provided by participants, this can be attributed to disagreement on wording rather than disagreement with the element.

To our knowledge, this is the first FRMS diagnostic tool that has been developed and validated using a modified Delphi process. Implementation of the FRMS diagnostic tool requires completion by Operational, Health and Safety, and Human Resources representatives of an organization with knowledge of the current state of their organization's FRMS, policies, and programs. A potential benefit of the tool is the structured design that systematically guides an organization to assess its FRMS against elements agreed by global experts in sleep science, chronobiology, and fatigue risk management. The FRMS diagnostic tool is comprehensive and considers all elements recognized by a panel of experts that, when effectively implemented, will support an organization's risk management approach to fatigue through an assessment of current controls [4,11]. Furthermore, the FRMS diagnostic tool will improve knowledge of, and increase an organization's understanding of the elements that require consideration, as part of a comprehensive systematic approach to reduce fatigue risk. Finally, the FRMS diagnostic tool will support organizations in identifying missing elements from their current approach that may subsequently inform the development of an implementation plan for improvement.

The overall effectiveness of an FRMS across all industries remains unclear. However, some studies have identified improvements in safety outcomes when elements of an FRMS are implemented, suggesting that complete implementation may also be beneficial [11]. For example, a randomized control trial to determine the efficacy of a fatigue risk management program for firefighters, that included education, sleep disorder screening, a napping policy, and the installation of black-out window coverings for sleep quarters, reported significant improvements in self-reported sleep quality as compared to the control group with no intervention [22]. However, given that the evidence on the elements required as part of a complete FRMS is unclear, it is first necessary to define these elements prior to assessing overall system effectiveness.

The potential benefits of utilizing an FRMS diagnostic tool to further improve an FRMS may reduce costs associated with sleep loss and include reduced absenteeism and staff turnover, increased productivity, and a reduction in compensation claims as a result of increased injury rates [8]. Although these potential benefits are well accepted, many organizations may not have the maturity to implement a comprehensive FRMS approach due to the potential financial and time investment required for implementation. Therefore, a compliance-driven approach may continue to be preferred by many organizations. An important consideration for the practical implementation of the FMRS diagnostic tool is the requirement for an organizational representative to self-assess their organization's current level of implementation. This requires a minimum level of expertise and knowledge whereby an organizational representative provides their subjective response. It is therefore recommended that following an organization's self-assessment using the FRMS diagnostic tool, a fatigue risk management expert reviews the results to ensure a quality assessment. This may present an opportunity for industry bodies and education faculties to develop training in fatigue risk management or incorporate it into existing health and safety training.

4.1. Limitations of this study

A limitation of this study is the potential biases from expert panel members, and the researchers as our modified Delphi process relied on their opinions. However, a strength of our study was the selection of participants who had extensive knowledge and expertise in sleep science, chronobiology, and fatigue risk management and the low attrition rate (12%) of participants. Therefore, the FRMS diagnostic tool will provide a valuable contribution to organizations in assessing their FRMS.

5. Conclusion

This FRMS diagnostic tool will be an effective method to assist an organization in systematically assessing their level of implementation of an FRMS and reducing risk. Through our modified Delphi process, we achieve high consensus from a diverse expert panel demonstrating that the elements, and their definitions are integral to an FRMS, and that the statements effectively assess the associated element. Future research should include a pilot of this FRMS diagnostic tool across a range of industries including mining, oil and gas, and healthcare to determine the FRMS diagnostic tool's reliability, usability, acceptability, and effectiveness. In addition, shift work organizations would benefit from further research into the effectiveness of the individual 14 FRMS elements in reducing risk to determine if specific elements should be prioritized and a weighted scoring system developed for the FRMS diagnostic tool.

Conflict of interest and declarations

Ian C Dunican is the Director and Chief Adviser of Melius Consulting Pty Ltd.

Acknowledgments

The authors would like to thank all the participants that have contributed to this research for their continued support and participation in our study. Also, to the funding programs the Australian Government Research Training Program and the Edith Cowan University Melius Consulting Industry Engagement Scholarship.

Author contributions

Gemma Maisey designed the study, collected the data, and drafted the manuscript. Marcus Cattani, Amanda Devine, and Ian C Dunican assisted in designing the study and editing the manuscript.

Funding

Gemma Maisey is supported by the Australian Government Research Training Program and the Edith Cowan University Melius Consulting Industry Engagement Scholarship to conduct this research.

Appendix A. Supplementary data

The following are the Supplementary data to this article: