An app for predicting nurse intention to quit the job using artificial neural networks (ANNs) in Microsoft Excel

Abstract

Background:

Numerous studies have identified factors related to nurses’ intention to leave. However, none has successfully predicted the nurse’s intention to quit the job. Whether the intention to quit the job can be predicted is an interesting topic in healthcare settings. A model to predict the nurse’s intention to quit the job for novice nurses should be investigated. The aim of this study is to build a model to develop an app for the automatic prediction and classification of nurses’ intention to quit their jobs.

Methods:

We recruited 1104 novice nurses working in 6 medical centers in Taiwan to complete 100-item questionnaires related to the nurse’s intention to quit the job in October 2018. The k-mean was used to divide nurses into 2 classes based on 5 items regarding leave intention. Feature variables were selected from the 100-item survey. Two models, including an artificial neural network (ANN) and a convolutional neural network, were compared across 4 scenarios made up of 2 training sets (n = 1104 and n = 804 ≅ 70%) and their corresponding testing (n = 300 ≅ 30%) sets to verify the model accuracy. An app for predicting the nurse’s intention to quit the job was then developed as a website assessment.

Results:

We observed that 24 feature variables extracted from this study in the ANN model yielded a higher area under the ROC curve of 0.82 (95% CI 0.80-0.84) based on the 1104 cases, the ANN performed better than the convolutional neural network on the accuracy, and a ready and available app for predicting the nurse’s intention to quit the job was successfully developed in this study.

Conclusions:

A 24-item ANN model with 53 parameters estimated by the ANN was developed to improve the accuracy of nurses’ intention to quit their jobs. The app would help team leaders take care of nurses who intend to quit the job before their actions are taken.

Key Points

• We performed ANN on Microsoft Excel, which is rare in the literature.

• An app was built to display results using a visual dashboard on Google Maps. The animation-featured dashboard was incorporated with the ANN model, allowing an easy understanding of the classification results with visual representations.

• The category probability curves were uniquely derived from the Rasch rating scale model and launched to the ANN prediction model to display the binary classification, using probability to interpret the prediction results.

1. Introduction

The World Health Organization predicts increased global demand for health and social care staff with the creation of 40 million new jobs by 2030^[1] and estimates a shortage of >4 million doctors, nurses, midwives, and others,^[2] far from meeting the rising demands for health care manpower, particularly for nurses in healthcare settings.^[3]

We observed that more than half of nurses (52%) are considering quitting the National Health Service (NHS) as work according to an exclusive survey by Nursing Standard and the Sunday Mirror.^[4,5] Approximately 21% of nurses (n = 44,082) reported intention to leave in the United States.^[6] In Turkey, approximately 43% had considered leaving nursing, 51% planned to leave their institutions,^[7] and 41% of nurses showed the intention to leave their jobs.^[8] In China, 71% of nurses intended to leave their jobs,^[9] and more than a third of pediatric nurses had the intention to quit their current jobs.^[10] In Australia, approximately 45% of nurses were unsure whether they would remain working in general practice.^[11] In Ethiopia, 78% (n = 79) of nurses had the intention to leave the current working unit of the emergency department or hospital,^[12] and 65% of nurses intended to leave their job,^[13] although only 20% of nurses intended to leave in Malaysia.^[14] In Iran, the high level of intention to leave nursing was expressed by 24%, and 25% had moderate intention.^[15] In Taiwan, nurses with low workplace justice had a higher intention of leaving the profession (adjusted odds ratio = 1.34, 95% confidence interval = 1.02-1.77) when compared with the nonintention group.^[16] All of these findings indicate that the phenomenon of nurses’ intention to quit their jobs deteriorates the supply of nurses in healthcare settings.^[3]

Lack of nursing staff may lead to increased medical incidence because of negligence, involving infection, falls, medication errors, tube dislodgement, pressure sores, and, most seriously, death.^[17,18] Nurses overloaded with a high number of patients may experience increased work stress, which has been identified as a key factor in increasing nurse turnover.^[19] The excessive workload on nursing staff may prolong patient hospitalization, increase patient morbidity and mortality, and increase the incidence of adverse events.^[20] Therefore, there is an urgent requirement for hospital management to predict nurses’ intention to quit the job and take preventive actions behind the insufficient nursing workforce.^[21]

Although numerous studies^[6,22-34] have identified factors related to a nurse’s intention to leave the unit, institution, and profession, no such techniques were found using machine learning (e.g., artificial neural network [ANN]) to predict nurses’ intention to quit the job in the literature. Advanced computer technology has enabled us to overcome the failure in prediction. Early identification of nurses’ intention to quit the job may prompt the involvement of supporting services to nurses for subsequently altering the outcome of lacking nurses.

Machine learning algorithms have been applied to predict nurses’ intention to quit the job, such as patterns of nurses’ electronic medical record (EMR) utilization using the Naïve Bayes classifier on the variable of voluntary turnover (VTO) for classification,^[35] multilevel logistic regression modeling to predict the intention to leave the nursing profession in eight countries (Germany, Italy, France, The Netherlands, Belgium, Poland, Slovakia, and China),^[36] the Multifactor Leadership Questionnaire (MLQ-5X) to assess nurses’ perceptions of their nurse managers’ leadership styles when the Anticipated Turnover Scale applied to assess nurses’ intention to leave the job for 280 nurses in 3 public sector hospitals and 1 university-affiliated (teaching) hospital in northern Jordan,^[37] and the Support Vector Machine applied to predict nurses’ intention to quit the job using working motivation and job satisfaction.^[38] Nonetheless, no studies of nurses’ intention to quit their jobs were found in the PubMed library using machine learning (or ANN). We are motivated to conduct the study using ANN to predict nurses’ intention to quit the job, particularly with an app that can help team leaders take the case of nurses who are going to quit their job before their actions are taken.

Accordingly, we hypothesized that the ANN model can be used to effectively predict the intention to leave and an app can be applied to predict the intention to leave in healthcare settings.

The aims of our study are to estimate the model’s parameters using the ANN model based on nurses’ responses to questionnaires on nurses’ intention to quit the job and design an app for assessment of nurses’ intention to quit the job.

2. Methods

2.1. Study sample and demographic data

Suppose the confidence level and intervals are set at 0.05 and ±5% and applied to the population of 300 novice nurses (approximately 30% of nurses in hospitals are ranked at the hierarchically advanced level [i.e., N0 for novice nurses, N1 for 1-year junior nurses, N2-N3 for ≥2 year experienced nurses, and N4 for senior nurses]^[39]) in a hospital, 169 participants are required to fulfill adequate sample size^[40-42] (i.e., input confidence level [ = 95%], confidence interval [ = 5], and population [ = 300] at the reference^[42]). We estimated the rate of refusal to respond to be approximately 40%. Therefore, the minimum number of participants for this study will be 282 (169/[1-0.4]) for each hospital.

In October 2018, we delivered 282 copies each to 6 medical centers in Taiwan, inviting 1692 ( = 282 × 6) novice nurses (i.e., nurse hierarchy at N0 and N1 only) to complete the 100-item questionnaire (from 5 domains in Fig. 1 and Supplemental Digital Content 1, http://links.lww.com/MD/G650) related to nurses’ intention to quit the job in previous studies.^[21-24] A total of 1104 nurses participated, with a return rate of 65.2% ( = 1104 ÷ 1692); see Supplemental Digital Content 2, http://links.lww.com/MD/G651), including 533 and 572 for Non-Intention and Intention, respectively, based on 5 items regarding leave intention; see the right-hand side in Fig. 1.

Figure 1. Feature variables extracted from the 100-intention-to-leave items regarding nurse intention to quit the job.

This study was approved and monitored by the NCKU Hospital institutional review board (06476734). All hospital and participants’ identifiers were stripped.

2.2. Study design

The psychometric properties of the 100-item questionnaire related to nurses’ intention to quit the job were mentioned in previous studies^[21-24]; see them in Supplemental Digital Content 1, http://links.lww.com/MD/G650. The study design was divided into 4 parts, including feature variables extracted from the data, model building and comparisons in accuracy and stability among 4 scenarios and 2 models, and 4 tasks achieved in the current study, as shown in Fig. 2.

Figure 2. The study flowchart using the 2 models and 4 scenarios to verify the featured variables available for the app for predicting nurse intention to quit the job.

2.2.1. Feature variables.

Feature variables were extracted from this 100-item questionnaire using logistic regression with a Type error set at 0.05, where the dependent variable (Intention as 1 and Non-Intention as 0) was determined by the k-mean clustering method^[43] on the summation scores of 5 items regarding leave intention; see the right-hand side in Fig. 1.

2.2.2. Model building and comparisons in accuracy and stability

2.2.2.1. The ANN model applied in this study. The ANN is a component of artificial intelligence that is meant to simulate a functioning human brain.^[44] ANNs are the foundation of artificial intelligence and solve problems that would otherwise be impossible or very difficult by human statistical standards.^[26] It was worth incorporating ANN (i.e., one of the famous deep learning methods) to see if it can improve the prediction accuracy on the classification of nurses’ intention to quit the job without directly asking questions of quitting the job.^[38]

2.2.2.2. Four scenarios and 2 models. Model accuracy (e.g., >0.7)^[39,41,43] and stability (or, say, generalization; e.g., the discrepancy between training and testing sets) were focused on several facets, such as model feasibility, efficacy, and efficiency. First, the 1104 participants were split into training and testing sets in a proportion of 70% to 30%, where the former was used to predict the latter.

2.2.2.2.1. Four scenarios. Four scenarios consist of 2 training and 2 testing sets derived from this grouping ratio: total cases (n = 1104) as a training set, its corresponding testing set (n = 300 ≅ 30%), another training set using 70% of participants (n = 804), and its corresponding testing set (n = 300). The higher and lower summation scores of nurses’ intention to quit the job were used in the training sets, while the middle summation scores of nurses’ intention to quit the job were used in the testing sets.

Second, the accuracy (e.g., sensitivity, specificity, area under the receiver operating characteristic curve, AUC) and stability (or generalization; e.g., using the training set to predict the testing set) were verified. The research data are shown in Supplemental Digital Content 2, http://links.lww.com/MD/G651.

2.2.2.2.2. Two prediction models. The ANN and convolutional neural network (CNN)^[26,44,45] were analyzed with the 4 scenarios mentioned above. CNN has traditionally been performed on Microsoft (MS) Excel,^[39,41,43] while ANN has not been paired along with MS Excel in the past. As demonstrated in Fig. 3 below, the ANN process involves data input in layer 1, where the data are joined with 2 types of parameters and run through the sigmoid function algorithms in layers 2 and 3. Finally, as shown on the right side and bottom of Fig. 3, the prediction model was optimized when the total residuals were minimized through the MS Excel function of sumxmy2 and solver add-in.

Figure 3. The process of estimating parameters in the ANN model. ANN = artificial neural network.

2.2.3. Three tasks achieved in this study

2.2.3.1. Task 1: feature variables and build the models. The featured variables were extracted from the 100-item questionnaire. The models were built and described in Supplemental Digital Content 3 with an abstract video, https://youtu.be/wDeBy3f4PHU.

2.2.3.2. Task 2: comparison of accuracies and stability across 2 models and 4 scenarios. The accuracy was determined by observing the higher indicators of sensitivity, specificity, precision, F1 score, accuracy, and AUC in the training sets of both models (i.e., ANN and CNN). The definitions are listed below:

True positive (TP) = the number of predicted Intention to the true Intention, (1)

True negative (TN) = the number of predicted nonintentions to the true nonintentions, (2)

False-positive (FP) = the number of nonintention minuses TN, (3)

False-negative (FN) = the number of Intention minuses TP, (4)

Sensitivity = true positive rate (TPR) = TP ÷ (TP+FN), (5)

Specificity = true negative rate (TNR) = TN ÷ (TN+FP), (6)

Precision = positive predictive value (PPV) = TP ÷ (TP+FP), (7)

F1 score = 2 × PPV × TPR ÷ (PPV+TPR), (8)

ACC = accuracy = (TP+TN) ÷ N, (9)

N = TP+TN+FP+FN, (10)

AUC = (1 − Specificity) × Sensitivity ÷ 2+(Sensitivity+1) × Specificity ÷ 2, (11)

SE for AUC = = ✓(AUC × (1-AUC) ÷ N), (12)

95% CI = AUC ± 1.96 × SE for AUC, (13)

The stability was determined by observing the AUC changes between the training and testing sets. The fewer AUC changes in a prediction model imply better stability. Comparisons of AUCs across 4 scenarios were made in both ANN and CNN models.

2.2.3.3. Task 3: app developed for predicting nurses’ intention to quit the job. A self-assessment app using participant mobile phones was designed to predict nurses’ intention to quit the job using the ANN (or CNN) algorithm with the model parameters. The result is shown as a classification and then appears on smartphones. The visual representation with binary (intention and nonintention) category probabilities was shown on a dashboard using Google Maps. This is because Google Maps provides us with the coordinate that enables us easy to lines and dots on a plate (or the cloud Google maps platform).

2.2.3.4. The online app provided to readers. We provided the app developed in this study to readers who can practice it on their own on the internet through the link in Supplemental Digital Content 4, App Online assessing nurse intention to quit the job at http://www.healthup.org.tw/irs/irsin_e.asp?type1=95.

2.3. Statistical tools and data analysis

IBM SPSS 19.0 for Windows (SPSS Inc, Chicago, IL, USA) and MedCalc 9.5.0.0 for Windows (MedCalc Software) were used to perform descriptive statistics, frequency distributions among groups, logistic regression analyses, and the computation of model prediction indicators mentioned in EQs from 1 to 13. The significance level of type I error was set at 0.05. ANN and CNN were performed on MS Excel (Microsoft Corp). Smart PLS^[46] was performed to interpret the domains of featured variables to the nurses’ intention to quit the job.

A visual representation of the classification was plotted using 2 curves based on the Rasch model.^[47] The study flowchart and the ANN modeling process are shown in Fig. 2 and Supplemental Digital Content 3, https://youtu.be/wDeBy3f4PHU, respectively.

3. Results

3.1. Demographic data of participants

The demographic data of the novice nurses (i.e., nurse hierarchy at N0 and N1 with nurse experiences less and greater than 1 year, respectively) are shown in Table 1. Two of the medical centers, B (n = 134) and F (n = 135), had fewer participants than the minimal satisfactory sample size of 169. Most participants were aged below 30years old, accounting for 91.4% ( = 1010/1104; see the bottom in Table 1), and worked for privately run hospitals ( = 60.4% = 667/1104). Normal distributions were seen in the selfassessment of health status and workload. Approximately 16.9% ( = 116/1104) identify themselves as being religious. The number of positive Intentions to leave ( = 571/1104 = 51.7%) is slightly higher than those with Non-Intention ( = 533/1104 = 38.3%). Only those variables of health status, work loadings, mental strain, and ages influence the intention to leave in demographic characteristics.

Table 1 .

Demographic data of the study sample.

		Intention to quit the job
Variable		No	Yes	n	%	x2 (Porb.)
A: Eligible sample size		533	571	1104	100
B: Medical Center						10.48 (0.06)
	Hospital A	85	107	192	17.4
	Hospital B	54	80	134	12.1
	Hospital C	93	113	206	18.7
	Hospital D	143	122	265	24.0
	Hospital E	86	86	172	15.6
	Hospital F	72	63	135	12.2
C: Hospital type						4.92 (0.07)
	Public run	229	208	437	39.6
	Private run	304	363	667	60.4
D: Religion type						4.75 (0.58)
	None	185	230	415	37.6
	General folk beliefs	253	249	502	45.5
	Buddhism	37	34	71	6.4
	Christian	28	32	60	5.4
	Catholic	6	4	10	0.9
	I-Kuan Tao	19	16	35	3.2
	Others	5	6	11	1.0
E: Self-assessing health score (the higher, the unhealthier)						6.62 (0.04)
	Low	144	127	271	24.5
	Medium	270	334	6	09	55.1
	High	114	107	223	40.2
F: Self-assessing work-loadings (the higher, the more loadings)						6.21 (0.04)
	Low	162	137	299	27.08
	Medium	264	299	563	51.00
	High	107	135	242	21.92
F: Self-assessing mental strain (the higher, the milder)						7.69 (0.02)
	Low	83	75	158	14.3
	Medium	314	382	696	63.0
	High	136	114	250	22.6
G: Age						2.23 (0.04)
	<30 years old	481	529	1010	91.5
	<40 years old	45	35	80	7.2
	<50 years old	7	7	14	1.3
The cutting point for the summation score of the intention to quit the job set at >11 using the k-mean clustering method.

3.2. Task 1: feature variables extracted from the study data

A total of 24 featured variables out of the original 100 items (Fig. 1 and Supplemental Digital Content 1, http://links.lww. com/MD/G650) were identified as statistically significant (P < .05) using the multiple logistic regression, with a threshold of the dependent variable (Intention) composed scores at <12 (Non-Intention) using the k-mean clustering method (Table 2).

Table 2 .

Eligible featured variables extracted from the data in this study.

No.	Featured variable	Prob.	Domain
1	Your religion and believing (0-7 automatically transformed)	<0.01	A6
2	Your age, yrs	0.03	A2
3	Self-assessing your physical ill (1-10 the more, the unhealthier)	0.09	A9
4	Self-assessing your work loadings (1-10 the more, the heavier)	0.02	A10
5	Self-assessing your mental strain (0-2 the more, the milder)	0.03	A11
6	My workplace supports me to participate in on-the-job education (0-6)*	<0.01	C3
7	Based on current job market conditions, my salary is appropriate (0-6)*	0.02	C9
8	I think my job is guaranteed (0-6)*	0.06	C10
9	I can take care of my work and family needs (0-6)*	0.03	C17
10	When I am at work, I can arrange family care (0-6)*	0.02	C18
11	I participate in the decisions made by my nursing supervisor (0-6)*	0.02	C26
12	I have the autonomy to make decisions about patient care (0-6)*	<0.01	C38
a13	I feel fatigued when I get up and have to face another day on the job (0-7)	0.01	D3
a14	I feel burned out from my work (0-7, the more, the more burnout)	<0.01	D5
a15	I feel like I am at the end of my rope (0-7, the more, the more burnout)	0.02	D8
b16	Easily understand how my patients feel about things (0-7, the less burnout)	0.02	D9
b17	I feel very energetic (0-7, the less, the more burnout)	0.01	D12
c18	I have become more callous toward people since I took this job (0-6)	<0.01	D18
c19	Overcoming pressure makes me stronger (0-6)	<0.01	B1
c20	I solve the problem, rather than let others make the decision (0-6)	<0.01	B3
c21	When changes happen, I can adapt (0-6)	0.03	B4
c22	I think I can control my life. (0-6)	0.01	B6
c23	No matter what obstacles are encountered, I achieve my goals (0-6)	<0.01	B10
c24	If necessary, I can do any with unwelcome way to influence others	0.02	B21
Intention to leave	1. I consider going to quit my current job		E1
	2. I investigated the none-nurse job opportunities°
	3. I investigated job opportunities in other hospitals		E3
	4. I like the current job and would like to stay at the nurse-related job (inverse)		E4
	5. I would like to stay at the current nurse professional job		E5
ab Frequent annually toward daily; c and NIQJ. from never to always.

^* P<.05

Four domains in 24 variables are presented in Fig. 4. R² in Intention to leave is 0.292. The most influential domain is burnout at work, with a path coefficient of 0.328, followed by quality of life (−0.239), demographics (−0.117), and resilience capacity (−0.083), indicating that higher burnout reflects stronger intention to leave. Other domains associated with Intention to leave are also significant in path coefficients, although the other 3 domains have a negative relationship with Intention, referring to their path coefficients.

Figure 4. Path analysis of 24 variables in 4 domains indicates that the higher burnout related to Intention to leave and other domains associated with Intention to leave are also significant in path coefficients using the 2 methods of SmartPLS and the module in MicroSoft Excel in panels A and B, respectively.

3.3. Task 2: accuracy and stability in comparison of models

When comparing the 2 models with the full data set of 1104 cases, the ANN model scored higher than the CNN model across all 6 indicators of sensitivity, specificity, precision, F1 score, accuracy, and AUC, suggesting that the ANN model had a higher accuracy.

The ANN model also performs better in terms of model stability when comparing the testing results with ACUs (e.g., 0.68 > 0.59 and 0.78 > 0.71 in Table 3 and the higher AUC of the testing-300 samples in Fig. 5).

Table 3 .

Comparison of statistics in models and scenarios.

Model		n	Sensitivity	Specificity	Precision	F1 score	Accuracy	AUC	95%CI
ANN
	a All cases	1104	0.83	0.81	0.82	0.83	0.82	0.82	0.80-0.84
	b Learning	804	0.86	0.85	0.88	0.87	0.85	0.85	0.83-0.88
	a Testing	300	0.76	0.64	0.54	0.63	0.68	0.70	0.65-0.75
	b Testing	300	0.80	0.77	0.65	0.72	0.78	0.78	0.74-0.83
CNN
	b All cases	1104	0.76	0.77	0.78	0.77	0.76	0.76	0.74-0.79
	a Learning	804	0.84	0.69	0.78	0.81	0.77	0.77	0.74-0.79
	a Testing	300	0.79	0.48	0.45	0.58	0.59	0.64	0.58-0.69
	b Testing	300	0.73	0.70	0.57	0.64	0.71	0.71	0.66-0.76

^a Using parameters in 804 learning cases to validate the results in the testing sample.

^b Using parameters in all 1104 cases to validate the results in the testing sample.

ANN = artificial neural network, AUC = area under the ROC curve, CNN = convolutional neural network.

Figure 5. Comparisons of AUC for the 2 models in panels A and B, respectively, along with 4 scenarios applied to the 2 prediction models (note that the 4 scenarios across 2 models are plotted and compared, in which the higher AUC was examined). AUC = area under the ROC curve.

We plotted the results: a receiver operating characteristic curves for these 4 scenarios and 2 models in Fig. 5. It is worth mentioning that the group consisting of 70% of the sample (n = 804) shown in Fig. 5 has the highest AUC (0.85) when compared with other scenarios owing to the higher discrimination power caused by selecting criteria using the lower and higher summation scores in intention to leave, as mentioned in the Methods section.

3.4. Task 3: app predicting nurses’ intention to quit the job

The interface of the app targeting novice nurses to predict intention to leave is shown on the left-hand side of Fig. 6. Readers are invited to click on the links^[36,37] and interact with the intention-to-leave app; see Supplemental Digital Content 4, App Online assessing nurse intention to quit the job at http://www.healthup.org.tw/irs/irsin_e.asp?type1=95. Notably, all 53 model parameters are embedded in the 24-item ANN model. Once responses are submitted, it generates a result as a classification of either possible Intention or Non-Intention without directly asking the question of quitting the job.

Figure 6. Snapshot of the intention-to-leave app on a smartphone referring to the links^[48,49] (note that the snapshot on the app is shown in the left panel and the assessment result in the right panel).

An example is shown on the right-hand side of Fig. 6, from which we can see that the participant scored a moderate probability (0.83) of nonintention, which is the curve starting from the top left to the bottom right corner. The sum of probabilities for Intention to leave and Non-Intention is 1.0. The odds can be calculated with the formula (p/[1-p] = 0.17/0.83 = 0.20), suggesting that this novice nurse has an extremely low probability or tendency to quit the job.

3.5. Task 4: creating dashboards on Google maps

Figure 6 is provided with links to references.^[48,49] Readers are invited to see the detailed information on the dashboard laid on Google Maps.

4. Discussion

4.1. Principal findings

We observed that the 24-variable ANN model can yield higher accuracy than the CNN model. An app was developed for predicting the intention to leave for nurses in Taiwan. As such, the hypotheses of the ANN model and app that can be used for predicting the nurses’ intention to quit the job were supported.

4.2. Comparisons to previous studies

Referring to previous studies,^[6,22-34] only the intention-to-leave factors were investigated. There has not been a predictive model built for analyzing nurses’ intention to leave with an online app. Although the authors developed models using the naïve Bayes classifier and support vector machine for predicting the intention to leave using nurse EMR utilization^[35] and working motivation, job satisfaction, and stress levels^[38] as predictors, no such app was demonstrated for readers to predict the intention to leave online as we did in this study.

More than half of nurses were considering quitting the job,^{[4,5,9,12,13]} similar to our finding: 51.7% and 38.3% for intention to leave and nonintention, respectively. Only those variables of health status, work loadings, mental strain, and ages influence the intention to leave in demographic characteristics (Table 1), similar to the study^[50] finding the age influencing the intention to leave. The dominant category is burnout at work based on the path analysis shown in Fig. 4. In the past 2 years, in the PubMed library, we have not seen research on the topic of nurses’ intention to leave related to health status, work loadings, and mental strain, which are worth further investigation in the future.

Although the intention to leave does not always lead to action (or behavior),^[51] predicting nurses’ intention to leave is an essential and necessary approach to establish an early warning mechanism from the perspective of human resource management^[38] when considering the constant shortage of nursing staff.^[1-3] In the current study, we verified that the ANN could improve the prediction accuracy on intention-to-leave classification, which is modern and innovative, where predictions are made without asking direct questions of quitting the job.^[38]

4.3. Implications and future work

The ANN performed better than the CNN in both accuracy and generalization. In this study, the sensitivity and specificity were improved. We have not seen others using the ANN approach to predict nurses’ intention to leave in the literature, which is a breakthrough made in this study. We have not noticed any article incorporating accuracy and stability to verify model feasibility, efficacy, and efficiency, but many authors have used the split scheme of a 70:30 ratio, invalidating their predictive models,^{[39,41,43,52]} and the model accuracy and stability were defined as AUC in training and testing sets, respectively, in a previous article.^[52]

As with the advancements in web-based technologies, mobile health communication is rapidly improving.^[53] There was no smartphone app designed to classify nurses’ intention to leave. Once the intention-to-leave model is executed, the classification system provides an early warning response for human resource management to react without directly collecting questions of the intention to leave.^[38]

4.4. Error analysis in the predictive model

As the quality control process emphasized the principle to consider more with the vital few and less with the trivial numerous,^[41,43] we suggest adapting with the matching personal response scheme for correct classifications in the model (MPRSA shown in Supplemental Digital Content 5, http://links.lww.com/MD/G652) and further increasing the accuracy toward approximately 100%^[43,52] in the predictive model. The reason is that the same response string will be matched by the MPRSA and lead to a correct classification if the responses are identical to that of the original dataset. We recommend searching individual answers in the dataset first. If found, assign the correct classification to this respondent. Otherwise (i.e., not found in the original dataset), the classification will be determined by the intention-to-leave model; see Supplemental Digital Content 5, http://links.lww.com/MD/G652.

4.5. Strengths of this study

ANN was performed on Microsoft Excel, which is rarely noticed in the literature. An app was designed to display classification results using category probability theory in the Rasch model.^[47] The animation-featured dashboard was incorporated with the ANN model, allowing an easy understanding of the classification with visual representations.

There are different types of algorithms for classification in machine learning,^[54,55] such as logistic regression, support vector machines,^[56] naïve Bayes, random forest classification, ANN, CNN,^[39,41,43] and the k-nearest neighbors algorithm.^[57] ANN was shown to be superior, with 93.2% classification accuracy in a previous study,^[56] similar to our results, although only 2 models (CNN and ANN) were compared. All research data and modules deposited in Supplemental Digital Content 6, All study datasets deposited at https://osf.io/pdza4/? view_only = 1471268e78244e87918f62675ae01882 help readers who are interested in this study replicate the study on their own, including partial least squares path modeling using MS Excel to analyze the featured variables shown in Fig. 4 in this study.

Furthermore, the curves of category probabilities based on the Rasch model^[47] are shown in Fig. 6. The binary categories (e.g., success and failure on an assessment in the psychometric field) have been applied in health-related outcomes.^[57-62] However, we are the first to provide the intention-to-leave model with the animation-type dashboard on Google Maps, as shown in Fig. 5.

4.6. Limitations and suggestions

Our study has some limitations. First, although the psychometric properties of the 24-item intention-to-leave assessment have been validated,^[21-24] there is no evidence to support that it is suitable for novice nurses in other countries/regions. We recommend additional studies using the same approach with the ANN or other models to estimate the parameters and explore the differences and similarities to this study.

Second, we have not discussed possible further improvements in predictive accuracy. For instance, whether other feature variables (e.g., variables not included in Fig. 1 and Supplemental Digital Content 1, http://links.lww.com/MD/G650) applied to the ANN model can increase the accuracy rate is worth discussion. It would be useful to look for other variables that can improve the power of the model prediction in the future.

Third, the study was carried out on the ANN model. Whether other predictive models have higher accuracy than the ANN is recommended for investigation.

Fourth, app results are shown on Google Maps. However, these achievements are not free of charge. For example, the Google Maps application programming interface (API) requires a paid project key for the cloud platform. Thus, the limitations of the dashboard are that it is not publicly accessible, and it is difficult to mimic by other authors or programmers for use in a short period of time.

Finally, the study sample was taken from novice nurses in Taiwan. The model parameters estimated for nurses’ intention to leave are only suitable for Chinese (particularly Taiwanese) health care settings. Generalizing these intention-to-leave assessment findings (e.g., the model parameters) might be difficult and constrained because the sample only took novice nurses working for inpatients into consideration. Additional studies are required to re-examine whether the psychometric properties of the intention-to-leave assessment are similar for inpatients and nurses in other workplaces.

5. Conclusion

The hypotheses made in this study have been supported, including the ANN being superior to CNN, and the app can be used for predicting the nurses’ intention to leave. We demonstrated the app and provided links for readers to practice it on their own, particularly observing the category probability curves based on the Rasch model.

The novelty of the app with our ANN algorithm improves the predictive accuracy of the nurses’ intention to leave. The integration of this app would hopefully help the nurse leader and human resource department to take care of nurses who are intended to leave their job before their actions are taken.

Acknowledgments

The authors thank Enago (www.enago.tw) for the English language review of this manuscript.

Author contributions

Hsiu-Chin Chen conceived and designed the study. Lifan Chen, Shu-Ching Ma, and Yu-Tsen Yeh performed the statistical analyses and were in charge of recruiting study participants. Tsair-Wei Chien helped design the app and interpreted the data. Huan-Fang Lee monitored the research. All authors read and approved the final article.

Conceptualization: Hsiu-Chin Chen, Lifan Chen, Shu-Ching Ma.

Data curation: Yu-Tsen Yeh.

Investigation: Huan-Fang Lee.

Methodology: Tsair-Wei Chien.