Predicting and Mitigating Freshmen Student Attrition: A Local-Explainable Machine Learning Framework

Abstract

With the emergence of novel methods for improving machine learning (ML) transparency, traditional decision-support-focused information systems seem to need an upgrade in their approach toward providing more actionable insights for practitioners. Particularly, given the complex decision-making process of humans, using insights obtained from group-level interpretation of ML models for designing individual interventions may lead to mixed results. The present study proposes a hybrid ML framework by integrating established predictive and explainable ML approaches for decision support systems involving the prediction of human decisions and designing individualized interventions accordingly. The proposed framework is aimed at providing actionable insights for designing individualized interventions. It was showcased in the context of college students’ attrition problem using a large and feature-rich integrated data set of freshman college students containing information about their demographics, educational, financial, and socioeconomic factors. A comparison of feature importance scores at the group- vs. individual-level revealed that while group-level insights might be useful for adjusting long-term strategies, using them as a one-size-fits-all strategy to design and implement individual interventions is subject to suboptimal outcomes.

Introduction

Machine learning (ML) methods, for a long time, have been known as black-box approaches with decent predictive accuracy but low transparency. Several approaches proposed in the literature (Carvalho et al., 2019; Gilpin et al., 2018) to interpret ML models and determine variables’ importance essentially provide high-level guidelines for practitioners based on the entire (average) sample characteristics. Whereas such guidelines could be informative and generalizable in contexts not involving humans as the decision-makers and role players (e.g., spam detection or classifying medical images) when it comes to predicting human behaviors and decisions, empirical generalizability of high-level one-size-fits-all interpretations is questionable. Particularly if such recommendations are to be used for designing and implementing individual-level intervention programs to change individuals’ mentality and persuade them not to act as predicted (by the ML model), it could lead to mixed results due to differences in variables’ importance at the group- vs. individual-level. Relying on a similar rationale, it has been shown in the biomedicine and healthcare analytics area that individual-level interpretations of the relative effect of features can result in more effective medical decisions (Ashley, 2016; Hodson, 2016). This has led to the emergence of a new area in healthcare analytics called precision medicine which generally aims to improve patient outcomes through individualizing diagnosis and treatment decisions (Kasztura et al., 2019).

Additionally, human decision-making is typically a holistic process. That is, people either consciously or unconsciously tend to assess a given situation from multiple various angles before finalizing their decision in that regard. Hence, to realistically mimic that process using ML algorithms, they have to be fed with comprehensive and multifaceted data.

College students’ attrition is one such decision-making process on which a large number of studies have been conducted using various statistical (Bai & Pan, 2009; Britt et al., 2017; Eng & Stadler, 2015; Zhang & Rangwala, 2018) and machine learning (Berens et al., 2018; Delen, 2010; Thammasiri et al., 2014) techniques to understand the contributing factors that lead students to quit college and help schools to take appropriate interventions. However, the factors identified in those studies resulted from analyzing data at the group level (i.e., across the entire sample of students), whereas at the individual level, each student who decides to drop out might have his/her reasons depending on his/her situation. Additionally, the focus in the prior studies is mostly on the personal (e.g., demographics), educational (e.g., GPA), and/or financial (e.g., loans) factors, while several socio-economic factors such as unemployment rate or income have been discussed in the literature as other likely influential considerations in making that decision. These factors, however, have never been studied using real-world data.

According to the US Department of Education, around half of the dropouts among undergraduate students of the same cohort occur in the transition from the Freshman college year to the Sophomore year. A more recent report from the National Center for Education Statistics maintains that while 32.9% of undergraduates nationwide do not complete their degree programs, around three fourth of them (i.e., 24.1%) drop out after their freshman year (Hanson, 2022). Hence a great deal of effort by the schools’ administrations must be devoted to ensuring the retention of freshman students. Relatedly, many of the student retention/attrition studies (including this study) have focused on first-year attrition/dropouts (i.e., the number of students not returning for the second year).

To address these gaps, the present study proposes a comprehensive hybrid framework by integrating state-of-the-art predictive ML methods and novel explainable machine learning approaches to improve proactive individualized human intervention programs. Specifically, we employ the proposed framework to address the freshman student attrition problem to help institutions gain individual-level insights on the students at risk of attrition as well as to design their individualized intervention. Such a framework will enable schools to (1) target the right students for offering their interventions, and (2) deliver the right type of intervention depending on the factors identified as highly contributing to each at-risk student. Particularly we show and discuss how different the insights obtained from interpreting ML models are at the group- vs. individual-level. We validate the proposed framework using one of the largest datasets ever used in the literature (containing 10 years of enrolled freshman students’ data from a midwestern university integrated with socio-economic factors obtained from government data sources).

The remainder of the paper is organized as follows. In the following section, we provide a summary of findings from the relevant prior studies on the student attrition problem. Next, we explain the proposed framework and provide details on the data set used for its validation. Finally, we present the results of the empirical analyses performed and discuss the implications.

Background

Retaining students, as the primary assets of a higher education institution, has been long a major concern. According to the National Center for Education Statistics1 of the US Department of Education, in 2019 only around 63% of full-time undergraduate students completed a bachelor’s degree at the same institution they had started in 2013 (i.e., 6-year graduation rate). This relatively high rate of dropouts, apart from the negative financial and budgetary consequences for post-secondary education institutions, raises major economic concerns resulting from the lack of qualified and trained applicants for existing job opportunities, particularly at the time of economic recessions (Restuccia & Taska, 2018). The world is currently anticipating a major economic recession as a consequence of the COVID-19 pandemic (Cardona et al., 2020). In addition, the decline in the birth rate in the United States between 2007–2012 resulting from the Great Recession has considerably decreased the number of anticipated college applications in the following decade; this will threaten the existence of many small- to mid-size institutions whose primary budget source is students’ enrollments.

There is no debate that freshman year is one of the most pivotal transition points for college students. Therefore, from the first to the second year of college student attrition, rather than overall retention and graduation rates, has become the central focus of researchers in recent years. The literature identifies first-year college success as a significant contributor to overall student retention and college graduation (Boyaci et al., 2015; Crawley et al., 2007; Reason, 2003; Roberts et al., 2009; Sidle & McReynolds, 2009).

Many researchers conducted studies that indicate a positive experience during the first year of college (e.g., student performance and GPA in the first and second semesters) are crucial predictors of student retention (Heywood, 2006; May & Chubin, 2003; Tan, 2002). In recent years machine learning techniques are becoming more popular and accurate in identifying college students with a higher risk of dropping out in the early stages and allocating the available resources based on their needs. In addition, these methods have been widely used for predicting and understanding the attrition of consumers in other business contexts such as e-commerce (Chou et al., 2021; Gattermann-Itschert & Thonemann, 2021; Höppner et al., 2020; Sabbeh, 2018; Tang et al., 2021; Vafeiadis et al., 2015), human resources (Alduayj & Rajpoot, 2018; Fallucchi et al., 2020; Qutub et al., 2021) or even in healthcare for blood donor retention (Kauten et al., 2021).

In a data mining study conducted on 16,066 students enrolled as freshmen during the 2004–2008 academic years, several classification models for predicting first-year student retention were used and through a sensitivity analysis, it was shown that factors like scholarships, loans, and fall GPA are the most important predictors of students’ retention (Delen, 2010). In another similar study, Delen (2011) compared three different prediction models for freshmen student attrition and concluded that educational and financial types of variables are significant predictors of student success. The study was conducted over an eight-year period. Artificial neural networks, decision tree (C5 algorithm), and logistic regression were the main classification methods used in this study to categorize students into “retained” and “not retained” with an approximately 80% accuracy obtained by ANN as the superior classifier.

Alkhasawneh and Hargraves (2014) developed a model composed of a qualitative and a quantitative study to predict first-year student retention for Underrepresented Minority (URM) students. Before developing their hybrid framework, the results of the genetic algorithm (i.e., quantitative model) and focus groups (i.e., qualitative study) were analyzed and incorporated into an ANN algorithm to predict first-year retention rates for students in STEM disciplines. The results from the ANN showed an overall classification accuracy of 74%, 79%, and 60% when using databases with all students, the majority of students, and under-represented students, respectively. High school rank, SAT mathematics scores, and mathematics placement test scores were considered strong predictors of retention.

In another study, Kondo et al. (2017) used records of 202 students available in the learning management system (LMS) during the first semester of 2015 to develop models using LR, SVM, and RF to predict GPA. Then, based on the deviation from the average GPA, they have defined a threshold for classifying students as “at-risk” or “not at-risk” of dropping out. The prediction from RF outperformed the others in terms of precision and sensitivity. Also, by tracking the comparative importance of explanatory variables every week, their model identified a ranking of important variables depending on the point in time (number of weeks after the semester started) analyzed.

In a recent work, Delen et al. (2020) proposed a Bayesian Belief network-driven framework to identify the critical determinants of student attrition and their conditional dependencies. They have showcased their proposed method using a feature-rich data set of college students to train a decent prediction model. Their sensitivity analyses show that students’ fall GPA, fall cumulative GPA, and the ratio of earned to registered hours in a fall semester are the most important determinants of their re-admission and registration in the following fall semester.

To adapt solutions to suit specific circumstances the majority of publications use the factors that lead students to leave their study programs at the “group level” (i.e., across the entire sample of students). In other words, much of the research and the development of practice has been based on investigating the “general” and “the most frequent” causes of student dropout. While the aforementioned practice provides insight into opportunities for interventional student advising, we argue that many of the factors associated with student withdrawal are not solely attributable to the student population as a whole; rather it is an individual matter. These can include aspirations and personal goals, expectations, values, and commitments as they relate to the choice of an educational direction. Given this degree of individual dependency, we chose the freshman attrition issue as the use case to present our proposed framework.

Additionally, our review of the literature of predictive and explanatory ML studies on students’ attrition revealed that the data used in those studies usually lack any features from one or more critical aspects that are discussed in theoretical studies as the main determinants of the attrition decision, namely demographics, educational, financial, and socio-economic factors. Table 1 summarizes some recent relevant studies, their subject group, the ML approach used, and the type of factors included in their analyses. Clearly, the socioeconomic factors are neglected in those research works whereas the theory suggests them as influential decision drivers (Tolliver, 2013; Tzafea & Sianou, 2018; Yorke & Thomas, 2003).

Table 1.

A summary of factors and approaches used in the literature for predicting attrition

Study	Subject	Approach	Factors Used for Prediction
			Demographics	Educational	Financial	Socio-economic
(Delen, 2011)	Undergraduate Freshman Students	ANN	X	X	X
(Berens et al., 2018)	Graduate students (German Universities)	AdaBoost	X	X
(Thammasiri et al., 2014)	Undergraduate Freshman Students	SVM	X	X	X
(Delen et al., 2020)	Undergraduate Freshman Students	Bayesian Belief Networks	X	X	X
(Lin et al., 2009)	Engineering college students	ANN	X	X
(Oztekin, 2016)	Undergraduate college students	SVM	X	X	X
(Dissanayake et al., 2016)	Undergraduate college students	Random Forest	X	X
(Fernández-García et al., 2021)	Undergraduate Freshman Students	Gradient Boosting Random Forest SVM Ensemble	X	X
(Cannistrà et al., 2021)	Undergraduate Sophomore Students	GLM DT Random Forest	X	X
(Baranyi et al., 2020)	Undergraduate Freshman Students	TabNet XGBoost Deep ANN	X	X
Current study	Undergraduate Freshman Students	Deep Neural Networks	X	X	X	X

Hence, to address these two major gaps, in the present study, we integrate state-of-the-art predictive and explainable ML approaches and propose a holistic framework that enables school administrations to take the best student-specific intervention action as it looks into the factors leading to one’s attrition decision concerning his/her specific academic, financial, and socio-economic situation. The following section details the proposed framework and the data used to validate it.

Materials and Methods

Data

To consider both university- and socioeconomic-related factors for studying student attrition, we combined data from four different sources for 10 years, between 2009 and 2018.

First, we used freshmen students’ enrollment data from a mid-western university in the United States. This data set contained more than 60 features indicating the demographics, educational information (both background and current), college life, and college financial status (loans, aids, etc.) of 39,470 unique freshmen enrolled in the period of study (only their freshman year data). Also, we derived an indicator variable showing whether each student enrolled for the Fall semester of their sophomore year to be used as the golden truth for their retention status.

Second, we obtained the US states’ annual GDP data from the Bureau of Economic Analysis website2 for each year in the period of study and mapped the GDP values to the students based on their state of residence and the year in which they enrolled as a freshman. GDP was used as a macroeconomic high-level proxy of the students’ economic situation each year as it was shown to be an influential factor in students’ enrollment in other countries (Andrianov, 2017).

Third, the average state-level per capita income was obtained from the American Community Survey of the US Census Bureau3 during 2009–2018 and mapped to the student records based on their state of residence and their freshman year. Income level has also been shown in the literature as one of the economic determinants of students’ college enrollment (Strickland et al., 1984).

Additionally, prior studies highlight the impact of the communities’ unemployment rate on college enrollment numbers (Dworak, 2020; Parolin, 2020; Strickland et al., 1984; Sutton, 2021). Hence, as the fourth data source, we obtained yearly state-level unemployment rate data from the US Bureau of Labor Statistics website4 and mapped them to the students based on the year and state of residence. Table 2 summarizes the key features and their statistics.

Table 2.

Descriptive statistics of the data set

Type	Feature	Mean (Std) or Proportions	Definition
Demographics	Age	18.51 (0.78)	-
	Gender	Male 48.1% Female 51.8% Other < 1%	-
	Race	White 71.3% Hispanic 6.6% Black 5.3% Mixed 9.1% Other 7.7%	-
	FirstGen	Yes 17.9% No 82.1%	Whether the student is the first generation in his/her family who entered college for a 4year degree.
	InState	Yes 71.3% No 28.7%	Whether the student is considered in-state for tuition purposes
Pre-College Info	HS_GPA	3.54 (0.38)	High School cumulative GPA
	YearsAfterHS	0.516 (0.769)	The gap between finishing high school and starting college (in years)
	ACT_Comp	24.84 (3.93)	Super ACT score (composite of all high subsections)
	Transfer_Credits	8.3 (10.3)	Total Credits brought in from high school or post-high school
College Info	ResLife	Yes 60.5% No 39.5%	Living in university housing during the first year
	GreekLife	Yes 28% No 72%	Participation in Greek life during the first year
	FALL_GPA	3.04 (0.85)	Cumulative GPA of the first Fall semester
	Total_Cred_Att	15.39 (5.29)	Total credits attempted during the first semester of college
	Total_Cred_Earn	12.43 (4.99)	Total credits earned during the first semester of college
	Pass_Ratio	0.85 (0.24)	A derived variable indicating the percentage of successfully passed credits during the semester of college
	College	Art Sci. 22.8% Eng. 18.9% Bus. 5% Agr.11.4% Other 31.9%
College Financial	AID_FALL	Yes 87.2% No 12.8%	Aid recipient in the Fall semester?
	FALL_PARENT_LOAN	Yes 16.7% No 82.3%	Parent loan recipient in the Fall semester?
	FALL_LOAN	Yes 40.2% No 59.8%	Loan recipient in the Fall semester?
	FALL_GRANT	Yes 80.7% No 19.3%	Grant recipient in the Fall semester?
	FALL_WORKSTUDY	Yes 7.5% No 92.5%	Workstudy recipient in the Fall semester?
	AID_SPRING	Yes 81.4% No 18.6%	Aid recipient in the Spring semester?
	SPRING_PARENT_LOAN	Yes 16% No 84%	Parent loan recipient in the Spring semester?
	SPRING_LOAN	Yes 37.3% No 62.7%	Loan recipient in the Spring semester?
	SPRING_GRANT	Yes 75.1% No 24.9%	Grant recipient in the Spring semester?
	SPRING_WORKSTUDY	Yes 8.6% No 91.4%	Workstudy recipient in the Spring semester?
Socioeconomic Factors	GDP	44,331.36 (4877.83)
	Unemployment Rate	5.06 (1.24)
	Per Capita Income	48,890.67 (5561.2)
Response Variable	Attrition	Yes 20.4% No 79.6%	Whether the student enrolled in the Fall semester of the sophomore year?

Methodology

Predictive Modeling

We employed a fully connected (a.k.a., dense) multi-layer perceptron (MLP) deep neural network approach to develop a model to predict students’ retention based on their personal, educational, financial, and socioeconomic factors. Even though some traditional tree-based classification algorithms (e.g., random forest, gradient boosted trees, ensemble trees) are deemed to be more transparent and easier to interpret than the neural networks, we made this choice due to the existence of several continuous numeric features in our data. While tree-based algorithms convert such features to threshold-based Boolean features and consider all instances on either side of the threshold the same, neural networks differentiate between instances even for small differences in their numeric features. In other words, they can extract more information from such features and utilize it for more accurate classification. This is especially important when the numeric features of the data are contextually important in determining the class of an instance, which is the case in this study (i.e., the high importance of college performance features as shown in the literature).

A deep MLP network, as a representation learning method (Goodfellow et al., 2016), combines simple features through its layers to derive complex features and then uses them for predicting the output (Zolbanin et al., 2020). Designing and fine-tuning a deep network’s architecture (i.e., the number of layers and units in them) typically requires a series of trials and errors combined with a decent understanding of the learning process. In addition to the number of layers and units, various other hyper-parameters (e.g., type of activation functions, initialization method, training batch size, number of training epochs, optimizer, learning rate, etc.) have to be fine-tuned for obtaining the best outcome. Technically, it is excessively computationally expensive (if possible, at all) to come up with a globally optimal combination of all the hyper-parameters for the network architecture. Hence, practitioners typically first manually determine some of the hyper-parameters (based on experience and familiarity with data and the subject matter), then employ approaches such as grid search (Larochelle et al., 2007), random search (Bergstra & Bengio, 2012), Gaussian process (Rasmussen, 2003), and sequential model-based optimization (Hutter et al., 2011) to optimize the rest over multiple trials.

As such, for this study after a few initial runs with some basic hyper-parameter settings, we decided to keep only 5 hidden dense layers in our architecture (see Fig. 1) and to employ a grid search approach to optimize the rest of the hyper-parameters. While there were only 34 input features, after performing some pre-processing (including binning some numeric features with sparse distributions and then one-hot-encoding all the categorical features) we ended up with 150 inputs to be fed into the network. The model involved a total of 30,088 trainable parameters (i.e., connection weights).

Fig. 1.

Deep fully connected MLP network architecture

Rectified linear unit (ReLU) was used as the activation function for each perceptron of the hidden layers. The primary advantage of ReLU over other common functions (e.g., linear or hyperbolic tangent), particularly for deep learning applications, is that it does not suffer from the vanishing gradient problem (Maas et al., 2013). The problem occurs when the derivation of the activation function for a given input (through the backpropagation of the error term in the gradient descent optimization process) becomes such small that no update will happen to the weights of such inputs through the training iterations. Also, given the binary nature of the response variable, a sigmoid transfer function was considered for the output layer.

To initialize the network’s weight we used the initialization schema proposed by He et al. (2015), which is known as the best schema for hidden layers including the ReLU activation function due to maintaining the variance of weights high during training and compensating for the variance loss because of the ReLU converting negative inputs to zero.

One of the common issues with the artificial neural networks’ training process is overfitting the model to the training data. The issue is particularly more likely to happen when the number of parameters (i.e., model weights) is considerably larger than the number of training instances to be used for adjusting them. Several strategies (e.g., decaying the learning rate, regularization methods, and adding dropout layers to the network, just to name a few) are suggested in the literature to handle the issue and avoid overfitting (Reed & MarksII, 1999; Srivastava et al., 2014). These strategies all work based on the idea of slowing down the training process and making a balance between the model performance on the training and validation data. Depending on the severity of the issue, one or a combination of multiple strategies can be used to address that.

For our model, we chose to use the learning rate decay strategy. It simply reduces the learning rate after the completion of each epoch. Such a strategy makes the network weights converge faster at the earlier training epochs and then gradually slows down their convergence as the model-predicted output becomes closer to the actual output values. Otherwise, the optimizer algorithm would quickly consume all noises to tune up the network weights and create a model that perfectly predicts the outcome for all the training instances. Various methods have been proposed for scheduling the learning rate decay, such as the exponential schedule (An et al., 2017; Li & Arora, 2019), non-monotonic schedule (Seong et al., 2018), and step-based schedule (Ge et al., 2019), mostly for complicated DL applications such as image or voice analysis. Given the relatively simple architecture of the MLP network a simple linear decay schedule, with a fixed decay rate, was used to avoid imposing any further, unnecessary parameterization to the model. Equation 1 indicates the employed linear decay schedule, where denotes the learning rate at epoch and represents the initial learning rate:

1

Given the imbalanced nature of the data (i.e., only around 20.4% attrition cases) and the bias caused thereby, regular classification practices are unlikely to produce decent per-class classification accuracy. Cost-sensitive learning (Kukar & Kononenko, 1998; Zadrozny et al., 2003) is a well-established approach to addressing the issue. It is essentially designed for classification problems where incorrectly classified instances of different outcomes (i.e., false positives vs. false negatives) involve different financial costs. As such, a higher cost of misclassification is considered for the more important class in the context to have the algorithm make fewer false classifications for that class (i.e., the minority class). Adopting the same logic to an imbalanced classification problem, we may assign a higher cost to the minority instances in the loss function to make up for the bias caused by the imbalance. Then to address the imbalance issue for our particular data set, we used a simplistic weighting approach by assuming an equal cost for false negative and false positive predictions. It means that given the 1,000:257 ratio of negative and positive cases (representing retention and attrition of students, respectively), the cost matrix shown in Table 3 was applied to the algorithm’s loss function.

Table 3.

Weights applied to the loss function of the classification algorithm

	Actual Negative	Actual Positive
Predict Negative	0	1
Predict Positive	0.257	0

Model Interpretation

Machine learning techniques have been generally known as “black box” methods with decent predictive capabilities, but little to no interpretability. However, recently there has been a stream of research works proposing various methods to address this shortcoming. These include approaches such as LIME (Ribeiro et al., 2016), NN-MCDA (Guo et al., 2021), and Layer-Wise Relevance Propagation (Bach et al., 2015), as well as some innovative strategies (Ferrettini et al., 2022) to make such methods more efficient5. They mostly aim at estimating the classic Shapley regression values (Shapley, 1988), that capture feature importance for linear models in the presence of multicollinearity relying on the cooperative game theory equations (Lundberg & Lee, 2017). Supposing each feature value of a given instance as a “player” and the prediction for that instance as the “payout” in a game, the Shapley value of that feature value determines the fair share of payout for each player(An et al., 2019).

Based upon the notion of Shapley values, Lundberg and Lee (2017) proposed a more intuitive method called SHaply Additive exPlanations (SHAP) to interpret complex ML models by weighing the marginal contribution of the feature values. The idea is that the outcome of each possible coalition/combination of features should be considered in determining the importance of a single feature. For a data set with N features, the method operates by training models, each with the whole data set and the same model hyperparameter settings, for all possible coalitions of features. Let’s say M1 and M2 are two prediction models trained using the same subset of features, except for one feature (f) which is present only in M2. For each given instance in the data, the difference between its predicted value by M1 and M2 is considered as the marginal contribution of the feature f for that particular instance. Considering all marginal contributions of f across all possible coalitions of features, we may calculate a weighted average of those marginal contributions and attribute it to the feature f as its importance score (i.e., SHAP value) in predicting the outcome of that particular instance. In mathematical terms, the SHAP value of feature f for instance x is (Lundberg & Lee, 2017) shown in Eq. 2:

2

where set represents any subset of features containing f, P_s(x) represents the predicted outcome for instance x using a model trained with the feature set s, |set| indicates the size of a feature set, and F represents the total number of features. The importance score for each feature represents the change in the expected model prediction when conditioning on that feature.

Essentially, one important difference between SHAP and the classic Shapley values approach is its “local accuracy” property that enables it to explain every instance of the data by calculating a single marginal contribution for that; whereas with Shapley values, a single overall importance score is assigned to the whole factor (Lundberg & Lee, 2017). In other words, the sum of features’ SHAP values for each instance yields the prediction for that instance in terms of deviation from the average predictions. Hence, in the specific context of this study, each freshman student’s attrition probability (prediction) can be explained in terms of the additive marginal contributions of her corresponding personal, educational, financial, and socioeconomic factors. Also, for each factor, the average of its SHAP scores across all instances (i.e., students) may be used as a proxy for its relative importance.

LIME is another popular approach for the local interpretation of ML models. It operates by building sparse linear models around each prediction to explain how the black-box ML models work in that local vicinity (Ribeiro et al., 2016). In essence, unlike SHAP, LIME does not exhaust all possible predictions for an instance using all possible feature coalitions. Hence, whereas LIME is more time-efficient in providing local importance scores for features in each instance, it does not guarantee consistency and local accuracy at the level offered by SHAP. Lundberg and Lee (2017) in their paper have discussed that LIME is essentially a subset of SHAP but lacks some properties. As such, we chose SHAP for the model interpretation part of our study.

In the last stage of our analyses, we employ the SHAP approach to understand the relationship among factors and interpret how their interplay leads a student to quit or enroll for the sophomore year. Figure 2 summarizes our proposed framework.

Fig. 2.

Summary of methods and procedures

Results

Prediction Model

To optimize the DL network’s hyperparameters we employed a grid search approach. That is while keeping the network architecture (i.e., number of layers, units, and activation functions) constant, we trained the model using multiple combinations of the other key hyperparameters (i.e., the optimizer type, learning rate, decay rate, batch size, and number of epochs). In each run, the selected optimizer was to minimize a customized (weighted) version of the binary cross-entropy loss (a.k.a, log loss) function. With the weight matrix shown in Table 3 applied, the weighted binary cross-entropy function used can be shown as Eq. 3:

3

where is the probability of class 1 (i.e., attrition) for a given instance i, is the actual class associated with instance i (1 if attrition, 0 if retention), and N is the total number of instances. When an instance belongs to class 1, the first part of the formula comes into play and the second part goes away, and vice versa. Note that a lower weight (0.257) is assigned to the loss resulting from a false positive prediction to compensate for the data imbalance, as discussed in the Methodology section.

Table 4 indicates the values used for each of the training hyperparameters in the grid search.

Table 4.

Grid search settings for hyperparameter optimization of deep NN model

Hyperparameter	Range of Values- step size	Number of values tested
Optimizer	[Adam, Nadam, RMSProp]	3
Learning rate	[0.001, 0.01]− 0.001	10
Decay rate	[0.01, 0.05]− 0.01	5
Batch size	[16, 32, 64, 128]	4
Epochs	[50, 75, 100]	3
	Total number of permutations	1,800

Since the grid search strategy does not exhaust the entire solution space, we then manually searched the vicinity of the obtained (nearly) optimal numeric hyperparameters by making slight changes in those values (one at a time) and observing the changes in the loss function. We realized that a learning rate of 0.0013 which is decayed at a rate of 0.005 led to a relatively steadier training process with a slightly better classification performance. So, we retained those values for the rest of the analyses.

Our best-trained deep network model with the mentioned settings achieved an overall accuracy of 88.4% in classifying students’ attrition/retention status in their sophomore year. The per-class accuracies (a.k.a., recall and specificity for positive and negative classes, respectively) of the model were 77.4% for the positive class (i.e., students who dropped out) and 91.1% for the negative class (i.e., students who enrolled). The area under the receiver operating characteristic curve (AUROC) of 0.886 suggests the remarkable power of the classification model in distinguishing the two groups of students.

Table 5 shows a comparison between our best results and those of similar recent studies. As discussed in the background section, this is the first data analytics study that considers socio-economic factors along with demographics, educational history, and college financial information to predict students’ attrition. As shown in Table 5, in most predictive performance areas, the present study outperforms similar recent predictive analytics efforts. Even though the study by Thammasiri et al. (2014) has reported considerably higher values for recall, specificity, and overall accuracy than every other related study, we believe that exceptional performance is, to a great extent, due to applying the trained model to a “balanced” validation data set in that study, whereas attrition data sets are naturally highly imbalanced with only 20–30% dropout cases, on average. The same argument applies to the relatively higher Recall value reported by Fernández-García et al. (2021) who used a roughly balanced original data set for training their model. Also, we argue that the considerably high predictive performance reported by Cannistrà et al. (2021) is, to a great extent, due to using sophomore-year students as the study subjects (i.e., predicting dropout in transition from sophomore to junior year, based on data from the first three semesters). While using freshmen data, prediction is based on only one semester of academic performance of the students, relying on a data set of sophomore students means having triple academic performance information as the former. However, as discussed earlier, between 50 and 75% of college dropouts occur after the freshman year and this makes the prediction of freshmen dropouts much more crucial from a practical standpoint.

Table 5.

Results and their comparison with similar past studies

Study	Subject	Approach	Factors Used for Prediction				Recall	Specificity	Accuracy	AUROC
			Demographics	Educational	Financial	Socio-economic
Current study	Undergraduate Freshman Students	Deep ANN	X	X	X	X	0.774	0.911	0.884	0.886
(Delen, 2011)	Undergraduate Freshman Students	ANN	X	X	X		0.685	0.938	0.812	NR
(Berens et al., 2018)*	Graduate students (German Universities)	AdaBoost	X	X			0.735	0.857	0.815	0.89
(Thammasiri et al., 2014)	Undergraduate Freshman Students	SVM	X	X	X		0.849	0.958	0.902	NR
(Delen et al., 2020)	Undergraduate Freshman Students	Bayesian Belief Networks	X	X	X		0.70	0.82	0.76	0.84
(Lin et al., 2009)	Engineering college students	ANN	X	X			0.420	0.788	0.717	NR
(Oztekin, 2016)**	Undergraduate college students	SVM	X	X	X		0.715	0.832	0.776	NR
(Dissanayake et al., 2016)	Undergraduate college students	Random Forest	X	X			0.625	0.954	0.859	NR
(Fernández-García et al., 2021)	Undergraduate Freshman Students	Ensemble	X	X			0.822	0.794	0.808	NR
(Cannistrà et al., 2021)	Undergraduate Sophomore Students	Random Forest	X	X			0.896	0.920	0.916	0.959
(Baranyi et al., 2020)	Undergraduate Freshman Students	Deep ANN	X	X			0.667	0.747	0.724	0.771

*From the several models involved in that study only the model reported for predicting dropout in sophomore year is shown in the table.

**This study predicts degree completion for 4-year college students.

Additionally, to indicate the contribution of socio-economic features (GDP, income per capita, and unemployment rate) to the predictive power of the model, we trained the model once more without those factors as the predictor. The results showed statistically significant declines (based on the two proportions z test) in all performance metrics; recall dropped from 0.774 to 0.724 (N = 2,052, p < 0.001), specificity from 0.911 to 0.875 (N = 5,842, p < 0.00001), accuracy from 0.884 to 0.844 (N = 7,894, p < 0.0001). Additionally, we compared the areas under the ROC curves using the method proposed by Hanley and McNeil (1983), which showed a statistically significant difference as well (0.886 vs. 0.876, p < 0.001).

Apart from an overall decent predictive performance, what is important from a practical standpoint is the decent performance of our prediction model in correctly identifying the students at risk of attrition (i.e., 77.4%). College administrative practitioners are constantly looking for that at-risk group to intervene appropriately and prevent them from giving up on their college journey. Schools typically consider various interventions including tutoring services, financial aids, and consultation services to be offered to such students depending on the problems inhibiting them from pursuing their degree. Besides correctly identifying such at-risk students, offering the right intervention to the right person is then another critical decision to be made by college administrators. The same intervention would not work for everyone; oftentimes, a wrong intervention may frustrate the student and make him/her not give another chance to the school for further investigating the issue and offering other interventions. This is discussed in further detail in the following section.

Model Interpretation

The SHAP package in the KNIME Analytics Platform was used to calculate the instance-specific feature importance scores. As discussed at the end of the previous section, one critical concern of the schools concerning the students identified with a high risk of dropout is to figure out the real cause leading them to such a decision and consequently deliver the right type of intervention to prevent that decision from happening. We argue that the local accuracy property of the SHAP feature importance scores (explained in the methods section) is the key to addressing this issue. While the probability of attrition calculated for two given at-risk students could be roughly the same, the marginal contribution of factors to those probabilities could be considerably different suggesting different issues leading those particular students to make a dropout decision.

Figure 3 indicates two waterfall charts showing the cumulative marginal contributions of the key factors for two students, both identified with a high risk of dropout, with roughly the same calculated probabilities (0.819 and 0.812) by the model. The first bar on the left in each chart represents the average probability across all instances, and the other bars show the positive or negative contribution of the relevant factors accumulated on that average to constitute the final calculated probability for each instance on the right. Also, the actual value of each feature is shown on the horizontal axes. For a better visibility, we have excluded the factors with a contribution smaller than 0.01 from both charts.

Fig. 3.

The marginal contribution of factors in calculating two close values for the probability of attrition (0.819 top vs. 0.812 bottom)

As shown on the x-axis, in terms of demographics both students are residents of the same state (TX) and are considered out-of-state for tuition purposes; they both are Hispanic and were 18 years old in their freshman college year (2014 and 2013, respectively), hence there was no gap between finishing high school and beginning college for either of them. The first student is a male who was the first generation in his family to attend college, whereas the second student is a female who has had at least one parent attending college in her previous generation. As shown in Fig. 3, there are considerably different patterns in the marginal contribution of factors determining their high odds of dropout as follows.

For the first student, the top three aggravating factors (increasing the probability of dropout) are Total Pass Ratio (= 0.31), Fall GPA (= Low), and ACT_English score (= 24) have contributed 38%, 10%, and 9% to the attrition probability, respectively. On the other hand, the income per capita (associated with the student’s state of residence), receiving financial aid in the Fall semester (AID_FALL), and having no gap between high school and college (YearsAfterHS = 0) are the top three factors reducing the chances of dropout for this particular student by 11%, 7%, and 6%, respectively. These numbers suggest that the main reason leading this student to drop out is his educational situation as opposed to financial issues or other potential causes and given his moderate high school GPA it seems that his low grades in the first semester of college must have more to do with his basic skills than the temporary emotional crisis. Therefore, probably educational interventions such as offering free tutoring sessions and consultation for planning the coursework should be more effective than offering additional financial aid. Particularly, given that the relatively low ACT English score (24 out of 36) of the student turned out to be the third mitigating factor, it is likely that assisting the student to improve his English skills (through the university writing center) and remove the possible language barrier could help retain him for the second year.

Similarly, for the second student, it seems like she had a relatively successful Fall semester in the first year with passing 14 out of 18 enrolled credits (i.e., Pass Ratio = 0.78) and a GPA categorized as “High”. However, as shown in the bottom chart, she has not enrolled for the Spring semester (i.e., SPRING_ENROLMENT = N) and has never returned after that for the next Fall either. Given these facts, educational interventions do not seem to be necessary and effective to retain this student. Likely, she did not enroll for the Spring semester due to financial issues (e.g., a sense of inability in paying off the student loan and/or not being selected for receiving financial aid in the spring semester) or other life crises. Therefore, interventions such as offering scholarships or additional financial aid, or free counseling sessions to assist her to tackle her life issues could be a decent course of action.

While individual analyses of the at-risk students could be much insightful in making the right decision at the right time for each student in the short term, having a big picture of the primary contributing factors across the entire sample could also provide invaluable insights for the schools’ administration to adjust their strategies over the long run. To this end, we calculated the mean absolute feature SHAP scores across all the students in the data. Figure 4 indicates the top 20 features with their importance scores.

Fig. 4.

Top 20 important features by their average SHAP scores

The chart suggests that, as expected, the student’s educational performance factors in the freshman year are the most critical in their decisions to drop out. Specifically, 3 out of the top 5 factors all have to do with students’ success in passing as many credits as possible with decent grades (GPA). This suggests that the main emphasis of the schools must be to ensure making a great educational experience for the students in the first year by providing decent planning consultation and promoting tutoring services.

Our data shows that around 94% of the students who did not enroll for the Spring semester of their first year, never came back in the following Fall and ultimately quit. While there might be various reasons for students not to enroll for the second semester, what seems to be critically important for the schools is to follow up with them early in that semester to ensure any problem that prevented them from enrollment will be resolved by the next Fall. Also, this suggests for a follow-up study to specifically focus on students with such profiles (i.e., enrolled in the fall, but not enrolled in the spring of the Freshman year) to better understand common reasons for such a large dropout ratio.

As indicated in Fig. 4, experiencing Greek life activities during the freshman year turned out among the top important factors. Our further investigation revealed that more than 89% of the students who experienced Greek life enrolled for the sophomore year. This sheds light on the unique importance of extracurricular activities in encouraging students to continue their education. Perhaps schools can provide such communities with extra funds and facilities to expand their activities and get more new students involved.

Interestingly, two of the three socio-economic factors (unemployment rate and GDP) incorporated in this study for the first time, turned out among the top 20 factors in terms of their average SHAP score. Specifically, our data show that the average unemployment rate associated with the students’ states of residence for those who dropped out is significantly higher than that of those who enrolled for the following Fall (5.14% vs. 4.78%, p < 0.01). Similarly, lower GDP is associated with higher chances of attrition (47,411 vs. 48,959, p < 0.01). In addition to the different socio-economic values for different states, another related issue is the schools’ different tuition policies for in-state vs. out-of-state students; Having the variable PR_STATE among the top features (and our further analysis on this feature) confirms that students categorized as out-of-state for tuition purposes are relatively more likely to quit. As such, a decently revised tuition policy seems to help decrease the attrition rate. Particularly schools may follow a state-specific policy for out-of-state students (as opposed to a one-size-fits-all policy) and consider relatively lower tuition rates for those coming from states with higher unemployment rates and/or GDP.

Apart from the insights that can be obtained from interpreting the prediction model at either group- or individual-level (for adjusting policies and designing individualized interventions, respectively), what emphasizes the importance of our proposed framework is the extent to which the group-level insights differ from the individual-level findings; also, how individual-level findings (and recommended intervention measures) vary from one person to another.

Post-hoc Analysis: The Effect of Cohort Year on the Retention Rate

While the cohort year factor was not originally included in our prediction model as a predictor, given that it potentially can be a proxy indicator of different state- or national policies that may affect retention rates, we performed some posthoc analyses to better understand the effect of that factor on the retention rate6.

First, we ran our prediction model again, this time including the cohort year as a predictor. The model performance did not show any significant improvement; in fact, we even observed a slight decrease in the performance metrics (%1–2), which we believe can be attributed to the additional dimensionality imposed by the new categorical predictor to the solution space. Hence, it seems that our original decision to exclude that predictor was not wrong, since its added complexity overweighs its informativeness.

However, we still wanted see whether our Freshman cohorts were consistent year on year in terms of their pre-college performance metrics, and otherwise how their differences affected their Freshman year performance as well as the corresponding yearly retention rate. To this end, we performed a series of statistical tests, for which we present a summary of the findings below.

Initially, we ran single-factor one-way ANOVA to compare different cohort years in terms of the key predictor factors. The results (at %95 confidence level) showed that for CLASS_RANK, PERCENTILE, SUPER_ACT_MATH, SUPER_ACT_READING, TOTAL_CREDITS_ATTEMPTED, and TOTAL_PASS_RATIO at least one of the cohorts has had a significantly different mean than the others. Given the different sample sizes across the cohorts, we then employed the Games-Howell test (Games & Howell, 1976) to perform pairwise cohort comparisons in terms of each of those factors and find out potentially problematic cohorts.

For half of the mentioned factors, we found only one pair of cohorts (the ones with the minimum and maximum means) whose means were significantly different, but the means were consistent across all the other pairs. However, in the case of SUPER_ACT_MATH, TOTAL_CREDITS_ATTEMPTED, and TOTAL_PASS_RATIO we had some interesting observations as follows.

For the SUPER_ACT_MATH factors the Games-Howell test revealed several inconsistencies among the Freshmen cohorts of different years. Particularly, there was a significant decline in the average Super ACT Math scores of the most recent cohorts (see Fig. 5).

Fig. 5.

The decline in the mean of SUPER_ACT_MATH score in the most recent cohorts

This suggests that either the academic institution under study has compromised its minimum admission requirements in recent years or the population of their applicants has declined in terms of their math abilities. These scenarios are in line with the trend we observed in the average high school ranks of the cohorts across years (Fig. 6), which shows a general increase in the high school rank of the admitted students over time. In other words, there seems to be a general decline in the quality of the educational background of the admitted cohorts in the most recent years at this institution.

Fig. 6.

Average high school rank of Freshman students in different cohort years

The other two factors for which we observed several inconsistencies across cohort pairs were TOTAL_CREDITS_ATTEMPTED and TOTAL_PASS_RATIO. We observed a significant decline in the mean of total credits attempted in the recent cohort years compared with the older Freshmen cohorts of the institution. However, we realized that (maybe as a result of this decline) the average total pass ratios have significantly increased in the recent cohorts compared with the older ones (Fig. 7).

Fig. 7.

The average total pass ratio of Freshman students in different cohort years

Because the overall Freshman retention rate at this institution has significantly increased over the period of study (ANOVA p-value < 0.0001, also see Fig. 8), the abovementioned findings tell an insightful story about how changes in the key characteristics of the cohorts over time have led to improved retention rate at this institution.

Fig. 8.

The retention rate of Freshman cohorts over time

In short, it seems like the school administration has found the misalignment between their school ranking and its admission requirements as a potential reason for the students leaving after the Freshman year. Accordingly, they adjusted their admission policy and offered admission to applicants with more aligned educational backgrounds with their standards. Given the compromised admission standards, accordingly, they started to advise incoming Freshman students to enroll in a relatively lower number of courses in their first semester to adjust their course load with their abilities. Even though that possibly has had financial consequences for the school (i.e., lower tuition and fees) in the short term, it has led to increasing pass ratios in those cohorts, which ultimately has increased the overall retention rate over time. This is particularly in line with the results of our SHAP analyses that suggest the Total Pass Ratio as the most impactful predictor of retention.

While not all details of the story induced from this post hoc analysis may be true (i.e., based on premeditated decisions or strategic plans), they still provide valuable insights for both research and practice. From a research standpoint, it suggests that factors such as alignment between the student’s level and school’s ranking, also between the student’s abilities and their course load can potentially be studied further as potential determinants of student retention. Also, from a practical viewpoint, revisiting the trajectory of events from this story suggests that continuous monitoring of the characteristics of the past cohorts and identifying potential reasons for their attrition can help policymakers to make impactful adjustments with long-term effects for the institution.

Summary and Conclusion

Student attrition has long been the number one concern for higher education institutions. Several studies have used various statistical and data mining methods for early identification of the students at risk of dropout and/or to understand the reasons for such decisions.

We rely on the framework proposed by Berthon et al. (2002) to elaborate our contributions. According to their framework, contributions of scientific studies can be described through four tuples, namely Problem, Theory, Method, and Context. Below, we have discussed our contributions through each tuple.

Problem Contributions

This study contributes to the long-standing problem of predicting college student retention in two ways. First, by improving the predictive performance of this problem through the use of a more comprehensive feature set (i.e., the largest longitudinal data set of freshman students in the literature), and state-of-the-art ML techniques. Second, the present study unboxes the so-called black-box model for the policymakers and school administrators, not just at a collective level, but also at the individual level to let them make the best decision concerning every single student to retain them.

We argue that one of the reasons that efforts for improving student retention rate across the nation are not so impactful is that most policies are made based on insights driven from data at an aggregate level, while many times students might decide to drop out due to factors that are neglected at that level and require more precise insights at the individual level.

In other words, we argue that while identifying students at risk of attrition is critical, the more important aspect of this issue, from a practical standpoint, is to take proper interventions to retain such students. While the former has been widely studied in the past literature, the studies focusing on the latter are limited to those that include only college-related features and/or employ a high-level analysis approach without accounting for the diversity of the students’ needs.

Theoretical Contributions

From a theoretical standpoint, the present study sheds light on the importance of previously under-studied socio-economic factors in determining students’ decision to drop out. Even though we used proxy variables at an aggregate level (i.e., GDP, average household income, and state-wide unemployment rate) to incorporate socioeconomic factors into our model, they turn out among the top influential factors in predicting retention.

Whereas all those factors were suggested in the literature to be theoretically related to the students’ attrition issue, we found no empirical evidence validating their actual importance. We believe that incorporating more precise individual-level features representing those factors in the future may lead to even more interesting insights about their role in one’s dropout decision.

This finding particularly suggests that educational institutions, in their strategic planning, should think beyond just the institutional factors and consider a higher weight for the societal and economic indicators. The implications could vary from adjusting admission and scholarship policies (i.e., more targeted policies) to developing plans to expand their community impact to improve the socio-economic factors in their corresponding communities. The latter may be realized through an advisor-advisee partnership with the industry to align their degree programs with real-world business needs, thereby enhancing the value of education in the eye of their students.

Methodological Contributions

We contribute to the method by (1) employing the state-of-the-art methods (i.e., deep learning and SHAP) to propose a hybrid framework for attrition prediction using demographic, college financial, socio-economic, college life, and college performance measures as well as evaluating their relative importance; and (2) integrating data from various college and non-college resources and introducing new measures into the problem. As argued by Shmueli and Koppius (2011) and a recent MISQ editorial by Miranda et al. (2022), the development and inclusion of new measures are one of the contributing roles of analytics research.

We encourage future researchers to extend our hybrid methodology by incorporating new features derived from various unstructured data sources (e.g., schools’ social media channels, content from students’ counseling sessions, and schools’ financial and educational policy documents) to gain more actionable insights for solving the problem.

Contextual Contributions

Finally, we contribute to the context of college student retention by looking at it through individual-oriented lenses. To the best of our knowledge, all previous studies on this problem try to prescribe one-size-fits-all suggestions for policymakers to help them reduce attrition rates. However, in this study we demonstrated that while aggregate-level insights are valuable, they are not enough for designing efficient policies, and the behavior of many students contradicts those insights, leading such policies not to be impactful, or even to backfire.

In other words, what works for most does not necessarily work for all, and this is potentially an even more severe problem when dealing with human subjects, given their complex decision-making process. Especially nowadays that educational institutions put a lot of emphasis on diversity and inclusion they should adapt their retention strategies accordingly by paying more attention to the students’ individual needs and values to be able to retain them through the end of their program. Accordingly, we argue that schools should leverage novel explainable ML tools and techniques and revise their student retention strategies by taking a more individual-oriented approach and offering at-risk students the type of help they need to keep attending school.

Limitations and Future Research

This study was only limited to analyzing the freshman college students enrolled in a single US institution. While within each cohort majority of dropouts occur in the transition from freshman to sophomore year, we encourage future researchers to use the proposed framework and evaluate the determinants of dropout (and possible changes in their pattern) for the later college years. Also, future research may take an experimental approach by performing a longitudinal study and validating the effectiveness of the individualized interventions facilitated by the SHAP analyses in comparison with the existing approaches that are being used to intervene. We also encourage future researchers to keep identifying more relevant factors from various data repositories and incorporate them in their studies to help ML algorithms better mimic the holistic approach used by humans for decision-making.

It is also evident that each institution is unique in terms of its culture, student population, local legislation, and several other aspects. Future research may apply the proposed framework to data collected from multiple institutions and study cross-institutional determinants of attrition about the institution-specific features.

Additionally, the student cohorts included in our data were all involved in the in-person modality of instruction before the COVID-19 pandemic. With the extensive exposure of students to the online and/or hybrid modalities as well as major policy changes during the pandemic, concerns might be raised regarding the generalizability of our findings for the post-pandemic era. We encourage future researchers to extend the present study by employing a comparative framework and including the instruction modality and other relevant factors to investigate changes in the attrition patterns and their determinants as a result of the global pandemic.

Finally, another general limitation of the student retention studies that use data analytics methods is neglecting the psychological and cultural factors that might influence one’s decision to drop out. We encourage future researchers to incorporate such factors either through a hybrid, analytics-behavioral, methodology and combining survey and transactional data, or by bringing in relevant features extracted from the university’s databases of advising and counseling services.

Biographies

Dursun Delen

is the holder of William S. Spears Endowed Chair in Business Administration, Patterson Family Endowed Chair in Business Analytics, Director of Research for the Center for Health Systems Innovation, and Regents Professor of Management Science and Information Systems in the Spears School of Business at Oklahoma State University. Prior to his academic tenure at OSU, he worked for a privately-owned research and consultancy company as a research scientist for five years, during which he led a number of advanced analytics research projects funded by federal agencies including DoD and NASA. Dr. Delen has authored more than 200 peer reviewed articles and 12 books and textbooks in the broad area of Business Intelligence, Business Analytics, and Data Science. He is often invited to companies for consultancy engagements and national and international conferences for keynote addresses. He regularly chairs tracks and minitracks at various business analytics and information systems conferences. Currently, he is the editor-in-chief for the Journal of Business Analytics and Ai in Business (in Frontiers in Artificial Intelligence), senior editor for the Journal of Decision Support Systems, Decision Sciences, and Journal of Business Research, associate editor for Decision Analytics and International Journal of RF Technologies, and is on the editorial boards of several other academic journals. He has been the recipient of several research and teaching awards including the prestigious Fulbright scholar, regents’ distinguished teacher and researcher, president’s outstanding researcher, and Big Data mentor awards.

Behrooz Davazdahemami

is an Assistant Professor of Information Technology and Supply Chain Management (ITSCM) at the University of Wisconsin-Whitewater. He received his M.S. degree in Industrial Engineering from the University of Tehran and his Ph.D. in Management Science and Information Systems from Oklahoma State University. His research interests include healthcare analytics, novel applications of machine learning, Information Systems (IS) privacy, and technology addiction. He is a member of the Association for Information Systems, The Institute for Operations Research and the Management Sciences, and the Decision Sciences Institute. Behrooz has published multiple research articles in journals such as Decision Support Systems, Journal of Business Research, Information & Management, and Journal of the American Medical Informatics Association (JAMIA). Also, he currently serves as an Associate Editor in the Journal of Business Analytics.

Elham Rasouli Dezfouli

is a Ph.D. student of Information Technology Management at the University of Wisconsin-Milwaukee. She has two M.S. degrees in Management Science and Information Systems from Oklahoma State University and Industrial Engineering from Isfahan University of Technology, Iran. Elham’s research interests include healthcare analytics, real-world applications of data analytics, and the role of cultural factors in information systems. She is a member of the Institute for Operations Research and the Management Sciences (INFORMS) as well as the Association for Information Systems (AIS). Elham has published research articles in the Journal of Healthcare Informatics Research as well as the proceedings of the American Conference on Information Systems (AMCIS).