The effect of COVID-19 on self-reported safety incidents in aviation: An examination of the heterogeneous effects using causal machine learning

Abstract

Introduction: Disruptions to aviation operations occur daily on a micro-level with negligible impacts beyond the inconvenience of rebooking and changing aircrew schedules. The unprecedented disruption in global aviation due to COVID-19 highlighted a need to evaluate emergent safety issues rapidly. Method: This paper uses causal machine learning to examine the heterogeneous effects of COVID-19 on reported aircraft incursions/excursions. The analysis utilized self report data from NASA Aviation Safety Reporting System collected from 2018 to 2020. The report attributes include self identified group characteristics and expert categorization of factors and outcomes. The analysis identified attributes and subgroup characteristics that were most sensitive to COVID-19 in inducing incursions/excursions. The method included the generalized random forest and difference-in-difference techniques to explore causal effects. Results: The analysis indicates first officers are more prone to experiencing incursion/excursion events during the pandemic. In addition, events categorized with the human factors confusion, distraction, and the causal factor fatigue increased incursion/excursion events. Practical Applications: Understanding the attributes associated with the likelihood of incursion/excursion events provides policymakers and aviation organizations insights to improve prevention mechanisms for future pandemics or extended periods of reduced aviation operations.

1. Introduction

On December 31, 2019, the Wuhan Municipal Health Commission reported a cluster of pneumonia cases from which a novel coronavirus was eventually identified (World Health Organization, 2020b). The first reported case in the United States occurred in Snohomish County, Washington, on January 23, 2020. The World Health Organization declared a public health emergency of international concern on January 31 (World Health Organization, 2020a). The following day, President Trump banned foreign nationals from entering the United States if they had been to China in the previous two weeks. The ban was broadened to all travel from European countries on March 11th (Presidential Proclamation No., 2020). By the end of March, approximately a third of the world population was subject to a form of lockdown. As early as January, U.S. aviation system-wide flight departures started to decline. By April, system-wide flight departures had decreased 67% from December 2019 and remained significantly below December 2019 levels through mid-2020.

Similarly, the number of reports submitted to the National Aeronautics and Space Administration (NASA) Aviation Safety Reporting System (ASRS) increased 70% in April 2020 from March 2020. The reporting reached a peak increase of 144% above the March 2020 level by July 2020. The increasing number of reports submitted to NASA's ASRS is counter to conventional wisdom. Flight departures were decreasing; by extension, airline operations should be less stressful, airspace less congested, and the reduced aircraft utilization should reduces maintenance demand. What safety and policy insights do the increasing number of reports provide for commercial aviation?

Understanding factors that induce increased reporting of safety issues is critical, as an accident or near accident events could lead to casualties. Researching the factors associated with “near misses” and their influence on reporting to ASRS improves our understanding of the causation and consequences of disruptive events to the aviation industry. Several studies build on these findings by analyzing the impact of decreased flight operations and bans in response to COVID-19 (COrona VIrus Disease of 2019) on particular study groups in the aviation workforce. The key aspects related to ASRS reporting include the psychological stress on pilots and cabin crew who are unemployed or on furlough (Alaminos-Torres et al., 2021, Widodo et al., 2021), air traffic controllers facing increased fatigue (Drogoul & Cabon, 2021), how the pandemic affected safety culture and climate in flight training organizations (Byrnes et al., 2022), and how airline management strategies can be formulated to respond to these uncertainties (Linden, 2021).

In this study, we investigate encompassing attributes of aviation incidents in the NASA ASRS and examine heterogeneity in the effects of COVID-19 in reporting “near-miss” events encompassing ground incursions and excursions in the controlled movement areas ramp, runway, and taxiway. 1 Hereafter, incursions and excursions occurring in the controlled movement areas ramp, runway, and taxiway will be referred to collectively as incursions/excursions. Our findings shed light on how heterogeneous effects of COVID-19 vary by flight attributes or characteristics and offer significant insights into identifying target attributes and formulating policies and strategies for reducing accidents in aviation. Furthermore, the attributes that are sensitive to an unprecedented situation, such as pandemics, in inducing “near-miss” events are important to accident prevention and aviation safety strategies. The study adopts a recently developed method, generalized causal forest (GCF), to examine the heterogeneous effects of COVID-19 and check the robustness of our results using difference-in-difference (DID) estimation.

The remainder of this paper is organized as follows. Section 2 provides background and reviews relevant literature. Section 3 discusses data and the empirical methods. Section 4 presents the results on the heterogeneous effects of COVID-19. Section 5 presents the conclusions.

2. Literature review

2.1. Aviation safety reporting system

The ASRS was created in 1976 by NASA to support the Federal Aviation Administration's (FAA) mission to improve safety by eliminating unsafe and preventable incidents. The ASRS is a voluntary, confidential, and non-punitive method for pilots, air traffic controllers, cabin crew, and maintenance technicians to report unsafe and hazardous situations (Chappell, 2017). The ASRS first-person reporting can reflect reporting biases, the degree of which cannot be measured (Hooley, 2018). For example, the number of reports filed for a type of incident represents an actual lower bound but cannot infer the prevalence of the incidents (Hooley, 2018). The ASRS reports are a valuable source of the “what“ and “why” related to safety incidents. The report data combined with the reporter's description of the “why“ provides critical insights regarding the direct and contributing factors affecting decision-making.

The first analysis of ASRS reports, published in 1976, analyzed 1,464 reports and recommended future studies related to human factors (Billings et al., 1976). Since its inception, the ASRS has processed 1,781,647 reports at an average rate of 5,471 reports per month (NASA, 2021). The constantly increasing number of reports and diversity of data types within the reports represent a challenge for researchers. Despite the challenges, the industry and academic analyses provide vital insights leading to enhanced aviation safety and policy.

In addition to increasing submissions, the reports' quantity and classification of recorded variables continue to increase. Presently, the ASRS report data consists of 125 variables, 87 of which are either multi-class with mutually exclusive classes (i.e., Flight Conditions) or multi-label representing different but related topics (i.e., Human Factors). The challenge is further compounded by acronyms and semantically different uses and degrees of importance of common and technical vocabulary. These challenges have led researchers to employ new and novel analytical techniques.

2.2. Aviation accident causal factors

Thanks to a long history of academic, industry, and government research, U.S. airlines have an enviable safety record (Pasztor, 2021). Researchers found there are a limited number of causal factors in accidents that manifest in response to scenario demands (Reason, 1990). The human factors analysis and classification system (HFCAS) identifies four sequential levels of barriers to adverse events; organizational influences, supervision, preconditions, and unsafe acts (Shappell & Wiegmann, 2000). The categorization of ASRS reports facilitates connecting causal factors to HFCAS levels. While reports identify causal categories applicable to each HFCAS level, the preponderance identify the situational and operator condition categories in preconditions level and decision, perceptual, and skill-based errors in unsafe acts level. Broadly these factors involve situational awareness, training, and stress. Situational awareness is an individual's ability to perceive, comprehend, and act upon environmental elements in current and future states. Personal and environmental stressors distract and negatively impact situational awareness. Degraded situational awareness has been demonstrated to negatively impact operator performance (Endsley & Kiris, 1995). Similar cognitive issues have been shown to affect operator performance by delaying or impeding takeover tasks (Agrawal & Peeta, 2021) and receiving inadequate training affected by emotional intelligence (Wang et al., 2021). An individual’s stress unrelated to job task or environment, has spill-over effects manifesting as error or delayed performance (Rowden et al., 2011). Krahnen et al. (2022) identifies stress mitigation activities specific to the unique environment of mobility operations. Identifying temporally relevant training and stress mitigation is a critical element in safe aviation operations. Training issues related to proficiency, currency, and experience are present in 40% of serious aircraft accidents (Kelly & Efthymiou, 2019). Improving and maintaining individual performance requires training and timely repeated feedback relevant to the causal issues (Komaki et al., 1980).

Several studies explored the effects of COVID-19 on safety concerns related to human factors, with a more specific focus on the workforce. Current studies examining the impact of COVID-19 on aviation identify adverse effects of the increased psychological stress levels on pilots and cabin crews (Alaminos-Torres et al., 2021). Cahill et al. (2021) surveyed and compared the levels of depression and anxiety in workers by subgroups (i.e., pilots, air-traffic controllers, cabin crews, engineering or maintenance, and others) without a statistical test to compare the heterogeneity in responses. A survey of 65 aviation engineering workers shows job and pandemic related stress negatively contributed to employee productivity. Further, the pandemic-related stress resulted in a more significant impact than job-related stress (Widodo et al., 2021). A key focus of this study is associated with examining the heterogeneous effects of COVID-19 by identifying characteristics that are sensitive to the event in reporting incursions/excursions. To the authors’ knowledge, there is currently no study that directly investigates the heterogeneous effects of the pandemic on aviation incidents that vary by encompassing characteristics of subgroups. The challenge of dealing with a large number of variables can be dealt with by applying a data-driven method to reduce the dimensionality of control variables.

2.3. Machine learning applied to the ASRS

Recent machine learning applications to analyze the ASRS dataset have identified and classified incident topics from the narrative statement field. Warp latent Dirichlet allocation (WarpLDA) was examined to determine if current topic modeling strategies are suitable for developing automated topic finding to ease manual workflows (Shi et al., 2017). Robinson et al. (2015) employ the method of latent semantic analysis to compare human-recorded causal factors of accidents in safety narrative. Researchers adopt an extended topic modeling, structural topic modeling (STM), to identify thematic highlights of the ASRS data (Kuhn, 2018, Rose et al., 2022, Paradis et al., 2021). A convolutional neural network framework, in combination with bidirectional long short-term memory neural network with attention mechanism (Att-BiLSTM) was also proposed to classify risk attributes in the ASRS data (Zhou et al., 2022). Machine learning and deep learning architectures were utilized to automate, identify, and validate potential unmanned aircraft systems safety risks (Abraham, 2022). Wallace and Ross (2006) utilized a transformer-based model based on RoBERTa, to classify aviation anomalies. These methods demonstrate promise to augment and eventually automate the identification of primary and contributing factors in ASRS reports. Ongoing research focuses on improving method accuracy. Shi et al. (2017) evaluated naïve Bayes, Hoeffding tree, and OzaBagADWIN algorithms to achieve accuracies ranging from 76% to 88% in the human factory ranging from 76% to 88% in human and aircraft-related factors. A Bayesian network to capture the causal impacts of risk factors was constructed by Zhang and Mahadevan (2021). Odisho et al (2022) use various machine-learning tools, including gradient boosting, decision tree, and support vector machine, to build a predictive model for runway excursions. Dong et al. (2021) utilized an attention-based long short-term memory model to achieve + 88% accuracy in identifying six of the primary human factors. The accuracies surpass the observed expert inter-rater agreement of 70% but fall short in the ability to consider the broad range of topic labels assigned in ASRS reports (Kierszbaum et al., 2021). This research highlights potential efficiency improvements to automate assigning labels to the single-label primary problem and multi-label contributing factors, a critical step to accelerating the process of processing reports into actionable policy information.

The application of causal machine learning has not been previously applied to identifying incident attributes in aviation safety reporting. The generalized random forest (GRF) technique enables nonparametric quantile regression, conditional average partial effect estimation, and heterogeneous treatment effect statistical estimation methods (Athey et al., 2019). The heterogeneous treatment effect estimation identified relationships between weather patterns containing numerous covariates and total factor productivity in agriculture (Stetter & Sauer, 2021). The GRF technique has been employed in crop rotation yields (Kluger et al. 2022), forest management policy (Miller, 2020), predicting preventable hospital readmissions (Marafino et al., 2020), and unemployment in early adulthood (Kuikka, 2020) studies. She et al. (2020) demonstrated improved speed-estimation accuracy of COVID-19 outbreak detection using GRF. Increasing empirical evidence demonstrates the effectiveness of GRF to estimate and infer causal relationships in a timely manner.

3. Data and empirical method

3.1. Data

Study data consisted of 7,246 incident reports from the NASA ASRS, covering January 2018 through June 2020, used to evaluate the potential influences of COVID-19 on reported incursions and excursions. The analyzed data were limited to January through June of each year to reduce the potential effects of seasonality. After excluding variables with high correlations, a total of 35 of 125 attributes were included in the analysis. Highly correlated variables were excluded to select a small number of influential variables to identify the heterogeneous effects and avoid masking influential variables from detection in the tree splits due to highly correlated variables (Bénard et al., 2022). In FAA parlance, incursions are the incorrect presence of an aircraft, vehicle, or person in a controlled movement area. Conversely, excursions occur due to the inappropriate exit from a controlled movement area. The dangers associated with airborne and ground incursions and excursions pose significant risks to all participants and are the most reported type of accident annually. Therefore, aviation operations rely on strict adherence to regulations and procedures to ensure high levels of participant safety.

The ASRS reports are composed of self-report data plus NASA expert assessment and characterization. Self-report variables for Operator, Reporter, Role, and Flight hours were considered in addition to the NASA categorizations of Anomaly, Human Factors, and Contributing Factors. The variable Operator refers to the commercial operators, air carrier (seating capacity > 60), air taxi (seating capacity < 60), corporate flight departments (two or more aircraft operated incidental to a business), fractional (shared aircraft ownership under 14 CFR 91 2), operating the aircraft at the time of the reported event. The Reporter variable refers to the official position of the individual who reported the incident. The reporter roles examined included the aircraft captain (directly responsible for the flight), first officer (assists the captain in the conduct of the flight), flight attendant (responsible for passenger safety), and air traffic control (ATC) (manage the flow of aircraft). The Role variable specifies which pilot was flying the aircraft. Finally, the Flight hour variable is the total flight hours for the individual who reported the incident.

NASA aviation experts assess the categorization of anomalies, human factors, and contributing factors. The categorizations include 20 anomaly factors, 11 human factors, and 6 contributing factors. The study utilized 9 anomalies, 11 human factors, and 4 of the contributing factor categorizations. The study focused on variables with correlations less than 0.4. The variables that include report attributes, incursion/excursions as anomaly outcomes, and the COVID-19 period as a treatment are summarized in Table 1 . Variables are transformed into binary variables to specify a particular attribute. Table 1 also includes the mean and mean difference of a variable by treatment, a COVID-19 period dummy. The correlation matrix of selected variables is available in Table 2 .

Table 1.

Summary statistics and mean comparison by pre-COVID and COVID period.

Categories of variables	Variables		Full sample		(1) Pre-COVID		(2) COVID		Difference	Chi-square stat
		Obs	Mean	SD	Mean	SD	Mean	SD
Treatment	COVID-19	7246	0.13	0.34
Outcome	Incursions/Excursions	7246	0.12	0.32	0.11	0.32	0.16	0.36	−0.04***	15.38
Operator	Air Carrier	7246	0.63	0.48	0.65	0.48	0.50	0.50	0.15***	75.29
	Air Taxi	7246	0.04	0.19	0.04	0.20	0.01	0.12	0.03***	16.25
	Corporate Flight Dept.	7246	0.04	0.20	0.04	0.20	0.04	0.19	0.00	0.05
	Fractional	7246	0.01	0.10	0.01	0.10	0.01	0.12	−0.00	0.69
	Other Operator	7246	0.01	0.08	0.01	0.08	0.00	0.06	0.00	1.06
Reporters	Captain	7246	0.39	0.49	0.41	0.49	0.32	0.47	0.09***	27.74
	First Officer	7246	0.17	0.38	0.18	0.38	0.13	0.34	0.05***	12.35
	Air Traffic Control Issue	7246	0.01	0.09	0.01	0.09	0.01	0.11	−0.00	0.83
	Flight Attendant	7246	0.04	0.20	0.04	0.19	0.07	0.25	−0.03***	19.89
	Pilot Flying	7246	0.42	0.49	0.41	0.49	0.44	0.50	−0.02	1.60
Contributing	Company Policy	7246	0.12	0.32	0.11	0.31	0.18	0.39	−0.07***	41.42
Factors	Human Factors	7246	0.59	0.49	0.58	0.49	0.66	0.47	−0.08***	23.56
	Procedure	7246	0.36	0.48	0.36	0.48	0.39	0.49	−0.03*	3.17
	Staffing	7246	0.02	0.13	0.02	0.12	0.04	0.19	−0.02***	20.74
Human	Communication breakdown	7246	0.26	0.44	0.26	0.44	0.27	0.44	−0.01	0.58
Factors	Confusion	7246	0.14	0.35	0.13	0.34	0.18	0.39	−0.05***	16.78
	Distraction	7246	0.12	0.33	0.11	0.32	0.19	0.39	−0.08***	43.26
	Fatigue	7246	0.02	0.13	0.02	0.13	0.01	0.12	0.00	0.11
	Human-Machine Interface	7246	0.06	0.24	0.06	0.24	0.05	0.22	0.01	1.26
	Other / Unknown	7246	0.03	0.18	0.03	0.16	0.06	0.23	−0.03***	22.41
	Physiological - Other	7246	0.02	0.14	0.02	0.15	0.01	0.08	0.02***	11.16
	Situational Awareness	7246	0.44	0.50	0.45	0.50	0.34	0.48	0.11***	38.99
	Training / Qualification	7246	0.09	0.29	0.09	0.28	0.10	0.31	−0.02	2.28
	Troubleshooting	7246	0.09	0.28	0.09	0.28	0.08	0.27	0.01	0.71
	Workload	7246	0.1	0.29	0.09	0.29	0.13	0.33	−0.04***	13.19
Anomaly	Air Traffic Control Issue	7246	0.2	0.40	0.21	0.40	0.15	0.35	0.06***	17.50
	Airspace Violation	7246	0.03	0.18	0.03	0.18	0.05	0.21	−0.02**	5.85
	In location: Inflight	7246	0.25	0.43	0.26	0.44	0.16	0.37	0.10***	44.36
	In location: Ground	7246	0.18	0.39	0.18	0.39	0.18	0.39	−0.00	0.04
Anomaly in Flight	CFTT/CFIT	7246	0.08	0.27	0.08	0.28	0.05	0.21	0.04***	16.49
	Loss of Control	7246	0.07	0.25	0.06	0.24	0.09	0.29	−0.03***	13.15
	Unstable Approach	7246	0.03	0.18	0.04	0.18	0.02	0.13	0.02***	8.90
	Wake Vortex	7246	0.03	0.17	0.03	0.18	0.01	0.08	0.03***	19.35
Pilot Flight	Flight hour in three months (below median)	7246	0.19	0.39	0.18	0.38	0.31	0.46	−0.14***	98.84
Hours	Flight Hour Total (below median)	7246	0.19	0.39	0.18	0.38	0.25	0.43	−0.07***	25.27
	Total Observations	7246			6296		950		7246

Note: *p < 0.10, **p < 0.05, ***p < 0.01. The minimum and maximum values are 0 and 1 for all variables as variables included in our analysis are binary.

Table 2.

Correlation matrix of selected important variables.

Variables	(1)	(2)	(3)	(4)	(5)	(6)	(7)	(8)	(9)	(10)	(11)	(12)
(1) Incursions/Excursions	1.00
(2) COVID-19 treatment	0.05	1.00
(3) Corporate Flight Dept	0.06	−0.00	1.00
(4) First Officer	−0.02	−0.04	−0.02	1.00
(5) Distraction	0.01	0.08	0.01	0.01	1.00
(6) Situational Awareness	0.09	−0.07	0.04	−0.01	0.14	1.00
(7) Training / Qualification	0.05	0.02	−0.00	−0.06	0.02	0.05	1.00
(8) ATC Issue	0.04	−0.05	0.03	−0.06	0.04	0.19	−0.00	1.00
(9) Ground	0.39	0.00	0.01	−0.06	−0.00	0.07	0.09	−0.06	1.00
(10) CFTT/CFIT	0.05	−0.05	0.05	0.02	0.03	0.17	0.05	0.17	−0.13	1.00
(11) Loss of Control	0.32	0.04	0.02	−0.04	−0.04	−0.01	0.10	−0.09	0.36	−0.07	1.00
(12) Flight hour total (below median)	0.13	0.06	0.01	−0.10	0.00	0.08	0.08	−0.08	0.16	−0.05	0.20	1.00

3.2. Generalized random forest

The empirical analysis involved multiple steps. First, the generalized random forest (GRF), a machine learning technique, was used to estimate the condition average treatment effects (CATEs) of the period of the COVID-19 pandemic for reporting the events of incursions or excursions. Second, attributes included in the NASA ASRS reports and subsets of these attributes most likely to experience incursions/excursions were identified. These attributes are the most important variables in predicting heterogeneity of the effects of the pandemic, which was an unprecedented event that significantly disrupted airline operations, thereby affecting factors related to flight safety. Third, a DID approach was employed for robustness. The DID examined the effects of COVID-19 on the likelihood of experiencing incursions/excursions using a set of covariates selected using the GRF technique.

To estimate the effects of COVID-19 that differ across subgroups, $Y_i$ was defined as the outcome variable for an individual $i$ who submitted the ASRS report of an incursion/excursion event. The individual is indexed by $i=1,2,\dots ,N$, and a vector covariates $X_i$. The COVID-19 pandemic as a treatment is indicated as a binary variable $T_i\in \left\{0,1\right\}$. An individual who submitted a report during the pandemic period was denoted as $Y_i\left(1\right),$ and $Y_i\left(0\right)$ if the report was made pre-pandemic period. $Y_i\left(0\right)$ is the control group. Only one of two outcomes was observed; therefore, the factual treatment effect is not identifiable (Holland & Rubin, 1987). The CATEs of an individual were estimated as the expected difference between the two potential outcomes conditional at $X_i=x$.

$$\tau \left(x\right)=E\left[Y_i\left(1\right)-Y_i\left(0\right)\left|X_i=x\right)\right]$$ (1)

The GRF method was adopted because it produces an “honest“ estimation based on recursive partitioning and subsampling of data (Athey et al., 2019, Zhang et al., 2022). Following the application in Athey and Wager (2019), estimation is considered honest by splitting data into two and taking the approach briefly explained below. Half of the observations are used to determine the variables to grow trees and the other half for CATE prediction and validation. The algorithm builds consistent and asymptotic estimators using an ensemble of trees, thereby used for statistical inference (Athey & Imbens, 2016).

Honest estimation is explained in applications of GRF in Carter et al., 2019, Zhang et al., 2022. A subset of observations $J$ is randomly drawn from $n$ number of observations without replacement, where $J$ is partitioned into two sets: $J_1$ to build trees that maximize the variance of CATEs estimation in a leaf and $J_2$ to estimate the CATEs. While building a tree, $m$ subset of covariates is used to split the subset of observations. Wager and Athey (2018) illustrated the procedures to satisfy honest conditions.

Athey et al. (2019) developed the GRF based on the above approach that performs in the presence of confounding and in estimating heterogeneous treatment effects (Nie and Wager, 2021, Robinson, 1988). The heterogeneous treatment effects are estimated as:

$$\widehat{\tau }=\frac{{\sum }_i{a}_i\left(x\right){(Y_i-{\widehat{m}}^{(-i)}(X}_i\left)\right)(T_i-{\widehat{e}}^{\left(-i\right)}\left(X_i\right)}{{\sum }_i{a}_i\left(x\right){(T_i-{\widehat{e}}^{\left(-i\right)}\left(X_i\right))}^2}$$ (2)

which is a semi-parametric approach to estimating $\widehat{\tau }$, where ${\widehat{m}}^{(-i)}$ and ${\widehat{e}}^{\left(-i\right)}$ is the out-of-sample prediction of a conditional outcome $m\left(x\right)=\mathbb{E}\left[Y_i\left|X_i=x\right)\right]$ and the conditional probability of being treated $e\left(x\right)=\mathbb{E}\left[T_i\left|X_i=x\right)\right]$. The term $a_i\left(x\right)$ is a data-adaptive kernel in which how often a unit $i$ in training has fallen into the same leaf at $x$(Athey & Wager, 2019). The procedures of GRF first estimate $\widehat{e}\left(X_i\right)$ and $\widehat{m}\left(X_i\right)$ separately, along with out-of-bag prediction. Residual treatment $T_i-{\widehat{e}}^{\left(-i\right)}\left(X_i\right)$ and outcome ${Y_i-{\widehat{m}}^{(-i)}(X}_i)$ are computed, and a GRF is trained on these residuals. Then ATE is estimated based on the following equation.

$$\widehat{\mathit{ATE}}=\frac{1}{n}\sum _i\left\{{\widehat{\tau }}^{\left(-i\right)}\left(X_i\right)+\frac{T_i-{\widehat{e}}^{\left(-i\right)}\left(X_i\right)}{{\widehat{e}}^{\left(-i\right)}\left(X_i\right)\left(1-{\widehat{e}}^{\left(-i\right)}\left(X_i\right)\right)}\left[{Y_i-{\widehat{m}}^{(-i)}(X}_i)-\left(T_i-{\widehat{e}}^{\left(-i\right)}\left(X_i\right)\right){\widehat{\tau }}^{\left(-i\right)}\left(X_i\right)\right]\right\}$$ (3)

Although GCF produces a consistent and asymptotic treatment effect estimation, we checked the robustness of our GCF results using a DID approach. We assume that unconfoundedness is satisfied as the exposure to COVID-19 occurred unexpectedly and is orthogonal to the potential observations $Y_i\left(1\right),Y_i\left(0\right)\perp T_i\left|X_i\right).$ We also assume that while air travel was restricted during the early stage of the pandemic, the flights or individuals were not selected based on the covariates (Rosenbaum & Rubin, 1983). In addition, we assume that overlap, known positivity or common support, is satisfied. The satisfied overlap means the probability of being exposed to the pandemic given a set of covariates is bounded to be less than 1 and greater than $0<Pr(T_i=1\left|X_i=x\right))<1,\forall x$. With the two assumptions, we can treat the observations as if they were generated from a randomized experiment and check the robustness of our GCF results using a DID approach.

3.3. Difference-in-difference estimation

For robustness of the results generated using the GRF, we extend our empirical analysis to estimate treatment effects (COVID-19) on incursion/excursion outcomes utilizing the DID approach for logistic regression. We aim to compare the DID coefficient for the heterogeneous effect of COVID-19 with the estimated results from GCF. The probability that incursion/excursion are reported is denoted as $P\left\{{\text{Nearmiss}}_i=1\right\}=P_i$ for an individual ASRS report $i$. The logistic regression model can be written as:

$${\text{logit}(P}_i)={\beta }_0+{\beta }_1{\mathrm{C}\mathrm{O}\mathrm{V}\mathrm{I}\mathrm{D}}_i+{\beta }_2{\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{e}}_i+{\beta }_3{\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{e}}_i\times {\mathrm{C}\mathrm{O}\mathrm{V}\mathrm{I}\mathrm{D}}_i+{\gamma \mathrm{X}+\epsilon }_i$$ (4)

where ${\mathrm{C}\mathrm{O}\mathrm{V}\mathrm{I}\mathrm{D}}_i$ denotes whether an individual NASA ASRS incursion/excursion was reported as a treatment in the COVID-19 time period. ${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{e}}_i$ indicates an attribute used as a conditioning variable selected under the GCF algorithm. We estimated the multiple models replacing ${\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{b}\mathrm{u}\mathrm{t}\mathrm{e}}_i$ with attribute variables. The coefficient of an interaction term, ${\beta }_3$, is our DID estimator for the heterogeneous effect that measures the difference in treatment effects between those whose $\mathit{Attribut}{e}_i=1$ and those with $\mathit{Attribut}{e}_i=0$ based on the following equation where $X_i$ represents the $\mathit{Attribut}{e}_i$:

$$\underset{({\beta }_0+{\beta }_1+{\beta }_2+{\beta }_3)-({\beta }_0+{\beta }_2)=({\beta }_1+{\beta }_3)}{\underbrace{[\text{E}(Y_i\text{|}{X}_i=1,T_i=1)-\text{E}(Y_i\text{|}{X}_i=1,T_i=0)]}}-\underset{({\beta }_0+{\beta }_1)-\left({\beta }_0\right)={\beta }_1}{\underbrace{[\text{E}(Y_i\text{|}{X}_i=0,T_i=1)-\text{E}(Y_i\text{|}{X}_i=0,T_i=0)]}}$$ (5)

The remaining selected attributes, except the one used to interact with the COVID-19 dummy, are treated as a set of control variables, denoted as $\mathrm{X}.$ The determinants considered in the logistic regressions are those listed in Table 4 , which include locations or flight anomaly situations, expert-assessed human factors, aircraft operators, and reporter roles. By controlling these subgroup characteristics, we consider unobserved fixed effects. The error term ${\epsilon }_i$ is expected to have a zero mean.

Table 4.

CATEs estimated using subgroup by binary levels of variable (Generalized random forest).

Categories of variables	Selected variables	Levels	CATEs	95% conf.low	95% conf.high	Difference in subgroup CATEs	T-stat	P-value
Operator	Corporate Flight Dept.	0	6.12	3.53	8.72	−17.78	3.91	0.00
		1	−11.66	−20.20	−3.12
Reporter	First Officer	0	4.04	1.56	6.52	8.07	−1.78	0.08
		1	12.11	3.55	20.66
Human Factors	Situational Awareness	0	2.30	−0.32	4.93	7.09	−2.60	0.01
		1	9.39	4.74	14.04
	Training / Qualification	0	4.91	2.24	7.59	5.43	−1.39	0.16
		1	10.34	3.17	17.51
	distraction	0	5.22	2.47	7.96	1.52	−0.45	0.65
		1	6.74	0.73	12.75
Anomaly	Air Traffic Control Issue	0	4.59	1.81	7.36	4.14	−1.24	0.22
		1	8.73	2.79	14.67
	In location: Ground	0	3.19	0.74	5.64	12.18	−2.74	0.01
		1	15.37	7.02	23.73
Anomaly in Flight	Loss of Control	0	5.46	2.93	7.98	−0.76	0.11	0.91
		1	4.70	−8.66	18.06
	CFTT/CFIT	0	4.39	1.98	6.80	13.01	−1.66	0.10
		1	17.40	2.26	32.54
Pilot Flight Hours	Flight Hour Total (below median)	0	5.67	2.83	8.52	−1.41	0.46	0.64
		1	4.26	−1.06	9.57

Our dataset involves 35 potential covariates at the beginning, including the characteristics of incursion/excursion situation attributes of those who reported the event. The numerous potential observed variables are a challenge for researchers when identifying the heterogeneity in the treatment effects. Therefore, the GRF method developed by Athey et al. (2019) was employed. The method extends the random forest nonparametric approach (Breiman et al., 2001). The GRF offers researchers a means to adopt flexible, functional forms to capture the heterogeneity in treatment effects of high dimensional data with a low computation burden (Dorie et al., 2019, Wendling et al., 2018).

4. Results: Heterogeneous effects of COVID-19 on near-misses

Overall, the estimation method of using the causal forest models succeeded in detecting heterogeneity in the effects of COVID-19 by subgroup characteristics. Table 3 shows a list of the top 10 important variables in order of how frequently each was used in creating trees. The variables, in order of variable importance, are listed by the weighted sum of how often the attributes were used to split in the forest (Athey et al., 2019). The quartiles of CATEs summarize the frequency of observations for each variable by levels. 3 The distribution of observations is also summarized based on CATEs in percentage terms. For example, 50.68% of the reports filed by corporate flight department operators fall in the lowest quartile (Quartile 1) of CATEs, while only 15.41% of corporate flight department operators’ reports fall in the highest quartile (Quartile 4), implying that more than half of corporate flight department operators were predicted to experience the lowest quartile of heterogeneous effects of the pandemic. First officer ASRS reporters have 47.51% of observations that fall in the quartile of the highest heterogeneous effect (Quartile 4). The ASRS reports, assessed by experts, with the human factors, situational awareness, and training as a cause of the incursion/excursion, also fall in the highest quartile of CATEs. The statistically significant p-values of the Wilcoxon Mann-Whitney test indicate that the distributions of estimated treatment effects are different across the levels (binary) of a variable. The selected important variables suggest that the impacts of the COVID-19 pandemic were greater in increasing incursions/excursions with certain characteristics that have a larger portion of observations fall in the highest quartile (Quartile 4) of CATEs in Table 3. For robustness, the observations were divided into subsets by levels of a variable and the causal forest estimation of the CATEs was performed separately on each subset. Table 4 presents the CATEs estimated separately over each variable’s subgroups (binary level). This extends the estimation in detecting the impact of COVID-19 heterogeneity by focusing only on the subgroups of characteristics. The statistically different estimated CATEs by subgroup supports the findings that the variables whose distributions are statistically different in Table 3 also experience heterogeneously diverging CATEs by a subset of attributes. Ground as anomaly location, the human factors situational awareness and training/qualifications, controlled flight toward terrain (CFTT) as anomaly flights, and being the first officer were the characteristics that experienced increased incursions/excursions due to COVID-19. The flights operated by corporate flight departments were sensitive to the pandemic in decreasing the reporting of incursions/excursions. Consistent with the distributions of CATEs using a full sample (Table 3), corporate flight departments experienced a negative impact of the pandemic in experiencing incursions or excursions. Fig. 1 shows the CATE and associated 95% confidence interval estimated separately over subsamples by binary level.

Table 3.

Observations by quartiles of conditional average treatment effects (CATE) of selected variables.

Categories of variables	Variables	Levels	Quantiles of CATE (number of obs.)					Quantiles of CATE (%)				Wilcoxon Mann–Whitney test
			1	2	3	4	Total Number	1	2	3	4	Z stat	P-value
Operator	Corporate Flight Dept.	0	1,664	1,747	1,777	1,766	6,954	23.93%	25.12%	25.55%	25.40%	10.58	0.00
		1	148	64	35	45	292	50.68%	21.92%	11.99%	15.41%
Reporter	First Officer	0	1,720	1,660	1,411	1,228	6,019	28.58%	27.58%	23.44%	20.40%	−24.27	0.00
		1	92	151	401	583	1,227	7.50%	12.31%	32.68%	47.51%
Human Factors	Situational Awareness	0	1,619	992	969	493	4,073	39.75%	24.36%	23.79%	12.10%	−38.04	0.00
		1	193	819	843	1,318	3,173	6.08%	25.81%	26.57%	41.54%
	Training / Qualification	0	1,754	1,656	1,646	1,530	6,586	26.63%	25.14%	24.99%	23.23%	−13.22	0.00
		1	58	155	166	281	660	8.79%	23.48%	25.15%	42.58%
	Distraction	0	1,671	1,582	1,541	1,554	6,348	26.32%	24.92%	24.28%	24.48%	−6.48	0.00
		1	141	229	271	257	898	15.70%	25.50%	30.18%	28.62%
Anomaly	Air Traffic Control Issue	0	1,733	1,433	1,300	1,347	5,813	29.81%	24.65%	22.36%	23.17%	−17.39	0.00
		1	79	378	512	464	1,433	5.51%	26.38%	35.73%	32.38%
	In location: Ground	0	1,775	1,724	1,605	825	5,929	29.94%	29.08%	27.07%	13.91%	−43.34	0.00
		1	37	87	207	986	1,317	2.81%	6.61%	15.72%	74.87%
Anomaly in Flight	Loss of Control	0	1,790	1,716	1,681	1,578	6,765	26.46	25.37%	24.85%	23.33%	−15.12	0.00
		1	22	95	131	233	481	4.57%	19.75%	27.23%	48.44%
	CFTT/CFIT	0	1,802	1,759	1,674	1,444	6,679	26.98%	26.34%	25.06%	21.62%	−21.72	0.00
		1	10	52	138	367	567	1.76%	9.17%	24.34%	64.73%
Pilot Flight Hours	Flight Hour Total (Below Median)	0	1,583	1,423	1,426	1,452	5,884	26.90%	24.18%	24.24%	24.68%	−5.09	0.00
		1	229	388	386	359	1,362	16.81%	28.49%	28.34%	26.36%

Fig. 1.

CATE estimated separately over subsamples (by binary level).

4.1. Difference-in-difference estimators for further robustness

The causal forest algorithm is beneficial to identifying heterogeneity in treatment effects because it reduces the number of variables to consider when there are numerous covariate candidates in a model. To further check the robustness of our results, the important variables selected in the GCF were used to estimate coefficients using DID logistic regressions. Table 5 summarizes DID marginal effect estimators by the subgroup attribute selected in GCF for both a naïve model without controls and a model with control variables. Consistent with the CATEs estimated in Table 4, the situational awareness and training/qualifications attributes, and reported by first officers, show a positive DID coefficient, indicating that flights with these attributes are more likely to report incursions/excursions due to the pandemic. The increase in the likelihood of experiencing and reporting the incidents is significantly greater for the attributes when the DID estimation shows statistically significant positive effects. A statistically significant increase of 4.63% in the likelihood of reporting the incursion/excursion event when the flight is categorized to have situational awareness issues and a 6.46% increase in the likelihood when assessed to be related to personnel training. The negative and significant coefficient of flights operated by corporate flight departments indicates that corporate flight departments are 14% less likely to report incursions/excursions due to the pandemic. Our estimation of DID coefficients in logistic regression using selected variables yielded results similar to the estimation of heterogeneous effects of COVID-19 using GCF.

Table 5.

Summary of difference-in-difference (DID) marginal effects estimates comparing Pre- and During-COVID-19.

Categories of variables	Variables	Model including control variables
		Pre-COVID	During-COVID	Effects*	P-Value
		(%)	(%)	(During-Pre), %
Operator	Corporate Flight Dept.
	0	10.88	15.61
	1	18.56	9.34
	Diff.	7.69	−6.28	−13.97	0.02
Reporter	First Officer
	0	11.2	14.53
	1	11.51	21.76
	Diff.	0.31	7.23	6.91	0.06
Human Factors	Situational Awareness
	0	9.84	12.06
	1	12.65	19.49
	Diff.	2.81	7.43	4.62	0.1
	Training / Qualification
	0	11.46	14.86
	1	9.67	19.54
	Diff.	−1.79	4.68	6.46	0.05
	Distraction
	0	11.23	14.89
	1	11.32	17.69
	Diff.	0.1	2.81	2.71	0.37
Anomaly	Air Traffic Control Issue
	0	10.23	13.61
	1	15.21	22.73
	Diff.	4.98	9.13	4.15	0.39
	In location: Ground
	0	5.96	8.22
	1	31.41	44.19
	Diff.	25.45	35.97	10.52	0.31
Anomaly in flight	Loss of Control
	0	9.28	13.62
	1	27.61	33.42
	Diff.	18.32	19.8	1.47	0.59
	CFTT/CFIT
	0	10.47	13.91
	1	21.64	35.02
	Diff.	11.17	21.11	9.94	0.23
Pilot Flight Hours	Total hours flown (below median)
	0	10.56	15.19
	1	13.47	16.93
	Diff.	2.91	1.75	−1.16	0.46

Note: * the effects indicate estimated coefficients for DID estimators.

Examining the distributions within the condition average treatment effect highlights the disparate nature of COVID-19′s impact on anomalous aviation events. The preponderance of variables revealed treatment effects skewed to higher quartiles. A notable exception is the heavy skew of corporate flight departments to the lowest quartile. The demonstrably different impact on corporate flight departments warrants further examination by human factors specialists. Two first officer and ground variables were significantly skewed to the highest quartile. The variables are notable but possibly conflated. As the junior officer in the aircraft, the first officer is disproportionately responsible for administrative tasks. While the treatment effects associated with the first officer should not be ignored, task assignment influences should be considered.

Similarly, the ground location variable treatment effects should be considered relative to the predominance of incursions and excursions occurring on the ground. The higher skewing for situational awareness, training/qualification, ATC, loss of control, and CFTT/CFIT indicate treatment effects warranting further scrutiny. The results indicate an additional notable exception in the minor treatment effects on total hours flown (below median). The commonly held belief is that less experienced pilots are disproportionately impacted by disruptions. The total hours flown (below median) result does not reflect the belief and warrants further examination by aviation operations and training specialists.

5. Conclusions, policy Implications, limitations, and future research

The application of machine learning to self-report safety databases increases the empirical evidence related to causal factors in aviation accidents. The goal of the current study was to examine the impact of an unprecedented period of reduced commercial aviation operations resulting from COVID-19 on incidents of aircraft incursions and excursions. The study demonstrated that a causal machine learning tool effectively analyzes subgroups within ASRS reports and identifies heterogeneous treatment effects associated with prolonged periods of reduced commercial aviation operations. The resultant method provides a targeted evaluation of temporal safety issues and training needs.

The key finding of this research is the prolonged period of reduced flight operations, and increased pandemic-related stressors created conditions conducive to aircraft incursions and excursions. The examination of NASA ASRS self-report data indicates the subgroups of aircraft operators, reporter roles, recency of experience, human factors, contributing factors, and anomaly were the most sensitive to the prolonged reduction in operations and experienced more incursions/excursions. The current study suggests the effect of COVID-19 differed across subgroups, expert assessed human factors, contributing factors, and anomalies. The most notable human factors were situational awareness, distraction, training, and the pilot's flight experience. The identification of causal human factors is not surprising. Recent research indicates COVID-19 created a foundation for human factors issues, with individuals more likely to experience worry, depression, and disinterest (Le & Nguyen, 2021).

The findings further suggest that safety awareness programs in aviation can incorporate identification strategies, such as causal machine learning tools, to identify focus subgroups or effective measures that reduce the impacts of prolonged reduced operations seen during the COVID-19 pandemic. Strategies should develop and improve support programs tailored to enhance situational awareness and aircrew training. Training programs should consider developing preemptive and targeted proficiency curricula to prepare and aid aviation personnel in skill retention. In coordination with training programs, the current study results can inform the design and deployment of policies to maximize the effectiveness of training strategies.

The application of the study results extends the available knowledge on aviation safety and issue identification methodologies. Although the preponderance of NASA ASRS reports occurred in the U.S. national airspace system, the international consistency in training standards, operational procedures, and regulatory structure suggest broad applicability of the findings.

While the study provided important insights regarding the application of machine learning in identifying causal factors, we encourage more research on applying new analysis techniques to extract the wealth of information contained within safety-related databases. These contributions advance academic research and improve the safety of vital transportation networks. Further research in this area should include:

First, the timely analysis of causal factors is critical. While ASRS reporting is voluntary and a lagging source, the significant increase in the first month COVID related reporting indicates the presence of safety issues and demonstrates a community desire for awareness. Further evidence is required to validate the application of machine learning to causal factor identification. Second, timely development and dispersal of training programs are required. The international aviation community benefits from numerous participants with the knowledge to rapidly develop training programs. It would be valuable to investigate a process for validating crowd-sourced training programs and disseminating them through existing regulatory and industry organizations.

Data Availability

NASA ASRS data can be downloaded from the Aviation Safety Reporting System website: https://asrs.arc.nasa.gov/. Codes are available upon request.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Dr. Youngran Choi is an Assistant Professor of Business Analytics at the David B. O’Maley College of Business at Embry-Riddle Aeronautical University. She holds a doctoral degree in Economics from Washington State University. Prior to the PhD program, she worked in business such as Ernst & Young and Deloitte and for international organizations such as UN FAO. Her research focuses on exploring causal inferences and factors to decision making. She employs econometric and analytic tools to examine data to understand personal and corporate behavior. Her recent publications include consumer preference for bio-based batteries in the Journal of Consumer Behaviour.

Dr. Jim Gibson is an Assistant Professor with the College of Business at Embry-Riddle Aeronautical University. Dr. Gibson is a retired United States Marine Corps naval aviator, squadron commander, and experimental test pilot. He is a powered-lift subject matter expert to the FAA’s Airmen Certification Standards working group. His research focuses on urban air mobility, energy economics, and sustainable development goals in East African carbon offsetting programs. Dr. Gibson is a US Naval Test Pilot School graduate who earned a Ph.D. in Systems and Engineering Management from Texas Tech University and an MBA from Duke University’s Fuqua School of Business.

Appendix A. An Explanation of the ASRS Variable Taxonomy

The ASRS is a voluntary, confidential, and non-punitive method for pilots, air traffic controllers, cabin crew, and maintenance technicians to report unsafe and hazardous situations (Chappell, 2017). The narratives and information within the ASRS reports describe the anomalous events, causal factors, categories of human factors, and quantitative information related to the participants and events. Presently, the ASRS report data consists of 125 variables, 87 of which are either multi-class with mutually exclusive classes (i.e., Flight Conditions) or multi-label representing different but related topics (i.e., Human Factors). The ASRS database online site details the ASRS coding taxonomy (https://asrs.arc.nasa.gov/search/database.html). The following tables provide an overview of the taxonomy, variable name, and description for variables considered in the study. Variables are transformed into binary variables to specify a particular attribute.

(See Table A1, Table A2, Table A3, Table A4, Table A5 ).

Table A1.

Derived variables utilized in the study.

ASRS Coding Taxonomy	Variable Name	Description
n/a	COVID-19 (treatment)	Incident reports occurring during the COVID-19 period.
Experience.Flight Crew: Last 90 Days	Flight hour (below median)	The number of flight hours flown by the pilot in the preceding 90 calendar days.
Experience.Flight Crew: Total	Flight hour total (below median)	The number of total flight hours flown by the pilot.

Table A2.

ASRS aircraft operator and participant function taxonomy.

ASRS Coding Taxonomy	Variable Name	Description
Aircraft Operator: Air Carrier	Air Carrier	The reporting flight was operated by an air carrier.
Aircraft Operator: Air Taxi	Air Taxi	The reporting flight was operated by an air taxi service.
Aircraft Operator: Corporate	Corporate Flight Dept	The reporting flight was operated by a corporate flight department.
Aircraft Operator: Fractional	Fractional	The reporting flight was operated by a fractional ownership operation.
Aircraft Operator: Other	Other Operator	The reporting flight was operated by an entity other than the previously listed categories.
Function.Flight Crew: Captain	Captain	The flight crew function of the reporting individual was the captain of the aircraft.
Function.Flight Crew: First Officer	First Officer	The flight crew function of the reporting individual was the first officer of the aircraft.
Function.Flight Crew: Pilot Flying	Pilot Flying	The flight crew function of the reporting individual was the pilot flying the aircraft.
Function.Air Traffic Control: ATC	ATC	The ASRS report was filed by ATC.
Function.Flight Attendant: Flight Attendant	Flight Attendant	The ASRS report was filed by a Flight Attendant.

Table A3.

ASRS Contributing factors, variable name, and description.

Factor Type	Variable Name	Description
Company Policy	Company Policy	Company policy was considered a contributing factor to the incident.
Human Factors	Human Factors	Human factors were considered a contributing factor to the incident.
Procedure (airspace authorization include here)	Procedure	A procedure was considered a contributing factor to the incident.
Staffing	Staffing	A staffing issue was considered a contributing factor to the incident.

Table A4.

Event anomaly types, variable name, and description.

Anomaly Type	Variable Name	Description
ATC Issues: All Types	ATC Issue	An anomalous event that occurred due to an ATC issue.
Airspace Violations: All Types	Airspace Violation	A violation of controlled airspace occurs when a pilot enters controlled airspace without a clearance.
Inflight Event/Encounter: All Types	Inflight	The anomalous event occurred when the aircraft was in flight.
Ground Event/Encounter: All Types	Ground	The anomalous event occurred when the aircraft was on the ground.
Inflight Event/Encounter: CFTT/CFIT	CFTT/CFIT	CFTT is defined as unintentional flight toward terrain. CFIT is defined as an unintentional collision with terrain while an aircraft is under control of the pilot.
Inflight Event/Encounter: Loss of Aircraft Control	Loss of Control	A Loss of Control accident involves an unintended departure of an aircraft from controlled flight.
Inflight Event/Encounter: Unstabilized Approach	Unstable Approach	An unstable approach is simply an approach that does not meet the criteria for a stable approach established by the aircraft operator.
Inflight Event/Encounter: Wake Vortex Encounter	Wake Vortex	Wake Vortex Turbulence is defined as turbulence which is generated by the passage of an aircraft in flight.
Ground Excursion & Ground Incursion	Incursion/Excursion (outcome)	The event was a ground excursion or incursion occurring on a ramp, runway, or taxiway.

Table A5.

Category of human factors, variable name, and description.

Category & Variable Name	Description
Communication Breakdown	Communication breakdown is defined as a loss of coordinated decision making between two groups or more that becomes a temporary inability to function effectively (Bearman, Paletz, & Orasanu, 2010)
Confusion	Confusion is when observed behavior is out of sync with the person’s mental model.
Distraction	Distraction occurs when anything reduces our focus on completing the current task.
Fatigue	Fatigue is a condition characterized by increased discomfort with lessened capacity for work, reduced efficiency of accomplishment, loss of power or capacity to respond to stimulation, and is usually accompanied by a feeling of weariness and tiredness (Salazar, 2007).
Human-Machine Interface	A human factors error assessed as a human–machine interface issue.
Other / Unknown	A human factors error was assessed as resulting from an unknown issue.
Physiological – Other	Aviation physiology is the physical and mental effects of flight on air crew and passengers.
Situational Awareness	Situational awareness is perceiving, understanding, and projecting the future of elements in a specified environment (Jones & Endsley,2000).
Training / Qualification	The systematic process of developing knowledge, skills, and attitudes; activities leading to skilled behavior (Martinussen & Hunter, 2017).
Troubleshooting	Troubleshooting is the process of identifying the cause and severity of a malfunction or discrepancy.
Workload	Workload is the mental demand placed on an operator.