Machine learning algorithms for prediction of measles one vaccination dropout among 12–23 months children in Ethiopia

Abstract

Introduction

Despite the availability of a safe and effective measles vaccine in Ethiopia, the country has experienced recurrent and significant measles outbreaks, with a nearly fivefold increase in confirmed cases from 2021 to 2023. The WHO has identified being unvaccinated against measles as a major factor driving this resurgence of cases and deaths. Consequently, this study aimed to apply robust machine learning algorithms to predict the key factors contributing to measles vaccination dropout.

Methods

This study utilised data from the 2016 Ethiopian Demographic and Health Survey to evaluate measles vaccination dropout. Eight supervised machine learning algorithms were implemented: eXtreme Gradient Boosting (XGBoost), Random Forest, Gradient Boosting, Support Vector Machine, Decision Tree, Naïve Bayes, K-Nearest Neighbours and Logistic Regression. Data preprocessing and model development were performed using R language V.4.2.1. The predictive models were evaluated using accuracy, precision, recall, F1-score and area under the curve (AUC). Unlike previous studies, this research utilised Shapley values to interpret individual predictions made by the top-performing machine learning model.

Results

The XGBoost algorithm surpassed all classifiers in predicting measles vaccination dropout (Accuracy and AUC values of 73.9% and 0.813, respectively). The Shapley Beeswarm plot displayed how each feature influenced the best model’s predictions. The model predicted that the younger mother’s age, religion-Jehovah/Adventist, husband with no and mother with primary education, unemployment of the mother, residence in the Oromia and Somali regions, large family size and older paternal age have a strong positive impact on the measles vaccination dropout.

Conclusion

The measles dropout rate in the country exceeded the recommended threshold of <10%. To tackle this issue, targeted interventions are crucial. Public awareness campaigns, regular health education and partnerships with religious institutions and health extension workers should be implemented, particularly in the identified underprivileged regions. These measures can help reduce measles vaccination dropout rates and enhance overall coverage.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• This study used large-scale, and nationally representative data.

• Unlike previous studies, the most powerful model interpretability tool (ie, SHapley Additive exPlanations values) was used to explain the direction and strength of the best-performing algorithm.

• A few important features (ie, knowledge of the vaccine) were not included due to unavailability.

Introduction

Measles is one of the most contagious diseases known worldwide. It is caused by the measles virus, which belongs to the genus Morbillivirus in the Paramyxoviridae family. This virus exclusively infects humans and there is no evidence of it affecting animals. Measles can lead to large outbreaks with high morbidity and mortality rates, particularly among vulnerable groups such as young children, pregnant women and individuals with weakened immune systems, including those with HIV, cancer or undergoing immunosuppressive therapy. Serious complications from measles can include ear infections, severe diarrhoea, blindness, encephalitis, pneumonia, and, in severe cases, death.^{1 2}

Africa hosted its first measles vaccination campaign with international attention in 1966. These campaigns demonstrated the effectiveness of vaccination against measles, even in the face of challenges like maintaining the cold chain for storing and transporting the heat-sensitive vaccine. Moreover, by May 1967, Gambia became the first nation in the world to stop the transmission of the virus.^{2 3}

When the WHO launched the Expanded Programme on Immunisation (EPI) in 1974, measles was one of the first diseases targeted for developing and implementing immunisation programmes globally. As a result of widespread childhood vaccination campaigns, global measles rates have significantly decreased. It is estimated that the measles vaccine prevented 31.7 million deaths worldwide between 2000 and 2020. The WHO now recommends vaccinating infants at 9 months in areas where measles is prevalent and at 12–15 months in other regions.^{3 4}

Due to the high infectivity of measles, a population must have at least 95% immunity to keep the disease’s effective reproductive rate below one and prevent epidemics. This raises the herd immunity threshold necessary for community protection. If the required immunisation rates, particularly for the first dose of the measles vaccine (MCV1), are not maintained, frequent outbreaks may occur when the disease is reintroduced.^{5 6}

The vaccine dropout rate, which measures the proportion of children who begin vaccinations but do not complete them, is a key metric used to assess the performance and continuity of an immunisation programme. The measles dropout rate can be calculated by comparing the number of infants who received the first dose of the pentavalent vaccine (Penta 1) to those who received the MCV 1. A dropout rate exceeding 10% indicates an issue with service utilisation, meaning that many children are not taking advantage of the available vaccination services.^{7 8}

Numerous studies have reported that sociodemographic factors are among the prominent determinants of measles vaccination dropout. These factors include the mother’s and father’s age, occupational status, educational level, residence and religion of the family. Health service utilisation and accessibility-related factors, such as antenatal and postnatal follow-up of the mother, place of delivery, distance to the health facility and the type of health professional providing maternal and child services, are also significant determinants.⁹,¹³

Despite the availability of a safe and effective measles vaccine in Ethiopia, recurrent and active measles outbreaks have been reported nationally since August 2021. The annual number of confirmed measles cases increased nearly fivefold from 2021 to 2023. Between then and May 2023, 16 814 laboratory-confirmed measles cases and 182 deaths, with a Case Fatality Ratio of 1.1%, were reported.¹⁴ According to the WHO and a meta-analysis conducted in the country, the primary factors contributing to the resurgence in measles cases and deaths are low population immunity and the number of unvaccinated individuals. Additionally, other contemporary outbreaks, conflicts, forced displacement and humanitarian crises that disrupt childhood vaccinations are further exacerbating the situation.¹⁴,¹⁶ This study, therefore, aimed to identify the major setbacks of the measles vaccination programme in Ethiopia through the application of robust machine learning (ML) algorithms using nationally representative data.

Methods and materials

Data source

This study used data from the 2016 Ethiopian Demographic and Health Survey (2016 EDHS) instead of the 2019 mini Ethiopian Demographic and Health Survey (2019 mEDHS). The primary reason is that the mEDHS lacks several important features necessary for predicting measles dropouts, such as the husband’s education, age, occupation, mother’s occupation, healthcare access, frequency of media exposure and the decision-maker on family healthcare needs. Additionally, the mEDHS included a relatively small number of children aged 12–23 months, who are potential recipients of the measles vaccine. Since ML algorithms perform best with larger data sets, the 2016 EDHS was deemed more suitable than the 2019 mEDHS for this study.¹⁷

The 2016 Ethiopian Demographic and Health Survey was conducted between 18 January and 27 June 2016. At that time, the country was administratively divided into nine geographical regions and two city administrations. The sample was designed to provide comprehensive estimates of key health indicators for the entire nation, as well as for rural and urban areas separately, and for each region and the two city administrations

The survey employed a residence-stratified sampling technique in two stages. Each of the nine regions was divided into urban and rural areas, resulting in 21 sampling strata. A total of 645 enumeration areas (EAs) were then selected, with 443 from rural strata and 202 from urban strata. Every household in the selected EAs was listed and documented. In large EAs with more than 300 households, the area was segmented, and only one segment was chosen. Thus, a cluster in the 2016 EDHS is either an EA or a segment of a large EA. Finally, a fixed number of 28 households per cluster was selected using a systematic random sampling technique. Proportional allocation and complete stratification were conducted at each lower administrative level

The questionnaires were administered to women and men aged 15–49 years who were either permanent residents or visitors who had spent at least one night in the selected household. Further details about the survey’s sample design can be found in the official survey report.¹⁷

Modelling software

This study used R programming language (V.4.2.1) to preprocess the data and run ML algorithms. Frequencies and percentages of features were described using descriptive analysis with the same software. Necessary R packages were downloaded from the Comprehensive R Archive Network, which is a network of File Transfer Protocol and web servers around the world that store identical, up-to-date versions of code and documentation for the R language.¹⁸

Study variables

Target and features

Measles dropout with a value level of 1 (=dropped out) and 0 (=received) was the target variable of the study. It was obtained from the difference in the number of 12–23 month-old children who were vaccinated for the first dose of the pentavalent vaccine (DPT-HepB-Hib) and dropped out for measles vaccination. Children who did not vaccinate for the first dose of the pentavalent vaccine (DPT-HepB-Hib) were not included in the study.^{19 20} There were a total of 26 features (both numeric and factor). It includes the mother’s age, region, residence, mother education, number of children, family size, Wealth Index, media exposure, mother’s smoking status, healthcare access, marital status, husband’s education, husband’s occupation, husband’s age, mother’s occupation, place of delivery, whether the child was checked within 2 months after delivery, place/facility where the child was checked, time of postnatal check-up of the mother, health professional who performed the postnatal check-up, religion, health insurance, CS-delivery, whether the mother earns more than the husband and the person who decides on the family’s healthcare need. The labels of the 26 features used in this study are available from the online supplemental table 1.

Data preprocessing

One of the most critical initial steps in building ML models is data preprocessing. This step is determined by the nature of the target variable and the features of the data set. Effective preprocessing can enhance the performance of any ML model. Key preprocessing tasks include data cleaning, target and feature engineering, managing missing values, handling imbalanced data, specifying resampling strategies and splitting data, though these tasks may vary depending on the type and nature of the data. If the target variable is continuous, it may be necessary to transform it, as this can improve predictions, especially in parametric models with specific assumptions. Target engineering techniques can be useful in such cases. Additionally, feature engineering is beneficial for nearly all ML models. Two crucial feature engineering tasks that most ML models can benefit from are categorical feature lumping and encoding.²¹

More features in a ML model frequently make it more difficult to understand and computationally inefficient. Additionally, few machine-learning algorithms are more resilient than others to less important predictors (such as tree-based models). An ML model will take more time to complete the more features we add to it. As a result, filtering non-informative features before running an ML model can significantly reduce training time and enhance prediction accuracy.²²

Data splitting into training and test (holdout) sets is crucial to understanding how well our algorithms generalise to new data. It contributes to making sure the best model is broadly applicable. A training set must be used for every task required to create an ideal machine-learning model. The test set will then be used to acquire an objective assessment of the performance of the final model.²¹

Imbalanced data can highly affect the ML model’s prediction performance. It happens when one class level of the target variable has very small instances than the other class level. Various data set sampling approaches are capable of balancing the target class. For most cases, a mixture of up-sampling and down-sampling, that is, the Synthetic Minority Over-Sampling Technique (SMOTE) is the most common and best approach.²³ The Model Architecture of the proposed prediction model is illustrated in figure 1.

Figure 1. Model architecture for machine learning algorithms for prediction of measles one vaccination dropout among 12–23 months children in Ethiopia. AUC, Area Under the Curve; DT, Decision Trees; GBM, Gradient Boosting Machines; KNN, K-Nearest Neighbour; LR, Logistic Regression; NB, Naïve Bayes; RF, Random Forest; SVM, Support Vector Machine; XGBM, eXtreme Gradient Boosting Model.

Machine learning algorithms

This study compared various supervised ML algorithms, including SVM, XGB, GBM, RF, DT, KNN, NB and LR, to identify the most effective one for predicting measles vaccination dropout among children aged 12–23 months in Ethiopia, based on their performance with loss functions.

XGBoost (eXtreme Gradient Boosting) is a powerful ensemble algorithm that builds multiple decision trees (DT) using gradient boosting. It minimises errors by iteratively adding trees that correct the mistakes of previous ones, capturing complex relationships in the data. Known for high accuracy, XGBoost efficiently handles large data sets and leverages multicore central processing units for scalability. The final model is a weighted combination of the trees, with weights assigned based on their contribution to improving accuracy.^{24 25}

Support Vector Machine (SVM) is a robust algorithm for classification and regression that maximises predictive accuracy while minimising overfitting. It identifies the optimal hyperplane to separate classes, using kernel functions to handle complex data. Parameter tuning is essential, often determined through cross-validation. SVM is easy to train, avoids local optima and scales well to high-dimensional data sets, though it relies on choosing an appropriate kernel.²⁶

Gradient Boosting Machines (GBM) is an ensemble method that combines weak learners, typically decision stumps, to enhance predictive performance. The algorithm sequentially adds models, each correcting the errors of the previous ones, while using gradient descent to minimise loss. GBM effectively handles various data types and produces interpretable models. However, the most resource-intensive part is finding the best weak learner at each step.²⁵

Random Forest (RF) is a popular ML algorithm for classification and regression that creates an ensemble of DTs trained on different data subsets. By averaging their predictions, it enhances accuracy and reduces overfitting. RF introduces randomness in feature selection at each split, promoting diversity among the trees, which generally leads to a more robust model.²⁷

DT is a non-parametric algorithm used for classification and regression tasks, structured hierarchically with nodes and branches. It recursively splits data based on feature values, selecting the best attribute at each internal node using criteria like information gain or Gini impurity. DTs handle both numerical and categorical data and adapt easily due to their automatic feature selection.²⁸

K-Nearest Neighbour (KNN) is a simple yet effective classification algorithm that classifies data points based on the classes of their closest neighbours. The ‘k’ parameter indicates how many neighbours to consider during the voting process. KNN is considered a ‘lazy learner’ because it stores the training data and performs classification only when needed. Finding the optimal value for ‘k’ typically requires experimentation.²⁹

Naïve Bayes (NB) encompasses a family of classification algorithms based on Bayes’ theorem, operating under the assumption that features are conditionally independent given the class label. This probabilistic approach estimates the likelihood of an instance belonging to a class, making it effective for classification tasks, despite the simplicity of its independence assumption.²¹

Logistic Regression (LR) is a statistical method for modelling binary outcomes by establishing a relationship between a dependent variable and one or more independent variables. It uses the logistic (sigmoid) function to transform linear predictions into probabilities between 0 and 1, facilitating effective binary classification.²⁸

Hyperparameter tuning and performance evaluation

Hyperparameter tuning optimises model configurations to improve performance and manage the bias-variance trade-off. In this study, tailored tuning strategies were applied to each ML model. To identify the best classifier, performance metrics like Accuracy, Precision, Recall, F1-score and Area Under the Curve (AUC) were compared.

Accuracy: defines the total number of correct predictions about all predictions made.

Precision/Positive Predictive Value (PPV): is a proportion that assesses how accurately the classifier separates cases. This metric is concerned with minimising the false positives.

Recall/sensitivity: this is a proportion that assesses the model’s ability to classify positive when the instance is an event/case.

F1-score: it is a harmonic mean of precision and recall. It assesses the predictive ability of a model by examining its performance on each class individually rather than considering overall performance like accuracy does.

Receivers Operating Characteristics (ROC)–AUC: a good binary classifier predicts well whether an event will or will not occur. Visualising this requires an ROC curve to be plotted. A ROC curve plots the false positive rate along the X-axis and the true positive rate along the Y-axis.

Model interpretability

Shapley values

SHAP (SHapley Additive exPlanations) are used to explain the output of a ML model. It is a method that explains how individual predictions are made by a ML model. It applies a concept in game theory used to determine the contribution of each player in a coalition or a cooperative game. Each SHAP value explains the deviation of the score for the query point from the average score of the predicted class, due to the corresponding variable. It applies primarily in situations when the contributions of each actor are unequal, but they work in cooperation with each other to obtain the outcome. For our classification model, Shapley computes Shapley values using the predicted class score for each class. Then, the Shapley values for the predicted class were plotted by using the ‘shapviz’ package. This study presented the two most commonly used global representations of SHAP values, the bar and Beeswarm plots.³⁰

Ethical approval and consent to participate

A formal online request was submitted to the DHS programme’s official website (https://www.dhsprogram.com/data) and ethical clearance and permission to use the data set were obtained. The data used in this study are publicly available, aggregated secondary data that has not any personal identifying information that can be linked to study households or individuals. The confidentiality of data was maintained anonymously.

Patient and public involvement

None

Result

Baseline characteristics

After data cleaning, a total of 3893 observations were used to build the ML models. The step-by-step process of data cleaning and extraction is displayed in online supplemental figure 1. About 1557 (39.9%) of children in the country had dropped out of measles vaccination. The majority of respondents belonged between the age group 15 and 30 years with a mean age of 28±6 years. About 840 (53.9%) children who dropped out of measles vaccination had a mother who had no education and 1481 (95.1%) of children who dropped out had a married mother. Three-fourths of 1192 (76.6%) of respondents whose child was dropped out were rural dwellers. More than half 933 (59.9%) of mothers whose child dropped out did not have any media exposure and about 703 (45.2%) of households with a kid dropped out of measles vaccination were classified under poor wealth index status. Regarding employment status, 970 (62.3%) of respondents with dropped-out kids do not work (ie, do not have a job). About 629 (40.4%) of fathers’ of dropped-out kids had no education, while only 137 (8.8%) had higher education and above (online supplemental table 2).

Data preprocessing results

Forty-nine records (1.26%) with missing values were excluded from the model building to enhance the quality of the ML models being built. Lumping was performed for all nominal features with many categories and/or small observations prior to the commencement of the model-building process. The feature ‘Media Exposure’ was generated by combining the frequency of reading newspapers, frequency of listening radio, frequency of watching television and frequency of using the internet. The feature ‘Healthcare access’ was created from whether it is a problem for the mother to get permission (from her husband) to go to a health facility or not, whether it is a problem for the mother to get money needed for treatment or not, whether the distance to the health facility is a problem for the mother or not and whether it is a problem for the mother to go alone or not. Zero and near-zero variance features were identified and removed. After this process, only 26 non-zero features were left and considered for ML model building. One-hot encoding was performed for the 26 non-zero variance nominal features, which expanded the number of features from 25 to 96. Ordinal encoding was performed for two features, that is (‘Educational status of the mother’ and ‘Educational status of the husband’).

The target variable of the study (ie, Measles dropout) has a class imbalance with a ratio of 1.5. Once the preprocessing step was over, this study handled the class imbalance issue using a stratified sampling technique of data splitting. Data were split into training (70%) and test (30%) sets using a stratified sampling technique. This resulted in 2724 training sets and 1169 holdout sets. In addition to that, an additional sampling method called SMOTE was considered during model building. This technique is a mixture of up-sampling and down-sampling and is the most common approach in ML models of the healthcare industry. This was critical, particularly, because the class levels of the target feature were not split into about 50/50.

Implementation of ML models

Eight supervised ML algorithms were implemented to classify the measles vaccination dropout status of 12–23 month-old children. Before the model training began, 10-fold cross-validation that was repeated five times was used as the resampling strategy to control the computational nuances of the ML models. This prevents overfitting as it fits each model on (10 minus 1) folds and makes one out of the 10 folds be used to compute the performance of the model. When this process is repeated five times, a different fold out of the 10 was treated as the validation set. This process is reported to increase the prediction performance of ML models in numerous articles.²¹ In addition, to compute performance metrics across resamples, the ‘summaryFunction’ argument was set to ‘twoClassSummary’.

A hyperparameter grid search was created and executed for models with more than one main hyperparameter. For the RF algorithm, hyperparameters such as the number of input variables (mtry), minimum size of terminal nodes (min.node.size) and the number of trees (Ntree) were considered in the grid search. For the Xgboost algorithm, the depth of the tree (max_depth), the minimum sum of the weight of all observations (min_child_weight), the number of variables supplied to a tree (colsample_bytree), the learning rate (eta) and max number of boosting iterations (nrounds) were tuned. Another ML algorithm with main hyperparameters to be tuned was GBM. For this algorithm, the total number of trees to fit (n.trees), step-size reduction (shrinkage), the maximum depth of each tree (interaction.depth) and the minimum number of observations in the terminal nodes of the trees (n.minobsinnode) were among the tuned main hyperparameters.

The tuned models of the Xgboost, RF, SVM and DT showed a significant improvement in their Accuracy and AUC compared with the default hyperparameters. KNN and GBM in contrast revealed a marginally better accuracy in their default values than the tuned. The optimal hyperparameter values and their corresponding performance metrics are summarised for each model in online supplemental table 3.

This study employed SMOTE to experiment with whether it could enhance models’ performance even in the absence of severe class imbalance. It provided a modest improvement in evaluation metrics, which supported its continued use in the analysis. This indicates that SMOTE can sometimes offer advantages beyond just addressing class imbalance, by potentially refining model evaluation metrics. The comparison of Evaluation Metrics before and after SMOTE is illustrated for each ML model in online supplemental figure 2.

Comparison of machine learning algorithms

The eight ML models were compared and evaluated, primarily, using their accuracy. Metrics like Precision, Recall, F1-score and AUC were also considered. Accordingly, the XGBoost model outcompeted the rest seven ML models with an accuracy of 73.9%, AUC value of 88.1%, recall of 93.7%, precision of 71.8 and F1-score of 81.3%. In contrast, the NB model reveals the lowest performance with an AUC value of 59.3%, accuracy of 60.0%, recall of 100%, precision of 60% and F1-score of 75.0% (table 1). The confusion matrix of the best model (ie, Xgboost) illustrates that about 1546 children who were predicted to drop out from measles vaccination were truly dropped out (ie, did not receive the vaccine), while 468 children who were predicted to receive the vaccine did truly received it. This makes our model more sensitive to detecting children with a higher probability of dropping out. In contrast, 104 children who were predicted to drop out by the model did receive the vaccine (ie, were false positives) (online supplemental figure 3).

Table 1. Comparison table of the eight machine learning algorithms used in this study.

ML model	Accuracy (95% CI)	SS/recall	Precision/PPV	AUC–ROC	F1-score	Kappa
Xgboost	0.739 (0.722, 0.755)	0.937	0.718	0.881	0.813	0.405
GBM	0.604 (0.586, 0.623)	0.709	0.020	0.646	0.038	0.017
RF	0.603 (0.594, 0.621)	1.000	0.601	0.820	0.750	0.008
LR	0.616 (0.598, 0.635)	0.861	0.632	0.627	0.729	0.122
SVM	0.703 (0.685, 0.720)	0.940	0.684	0.644	0.791	0.317
NB	0.600 (0.581, 0.618)	1.000	0.600	0.593	0.750	0.001
KNN	0.617 (0.599, 0.636)	0.918	0.623	0.620	0.742	0.096
DT	0.630 (0.612, 0.648)	0.880	0.639	0.597	0.741	0.150

AUCArea Under the Curve DTDecision TreesGBMGradient Boosting MachinesKNNK-Nearest NeighbourLRLogistic RegressionMLMachine LearningNBNaïve BayesPPVPositive Predictive ValueRFRandom ForestROCReceivers Operating CharacteristicsSSSensitivitySVMSupport Vector Machine

According to the AUC comparison of ROC curves, Xgboost presented the highest area (0.881), followed by RF (0.820) and SVM (0.644). The ML model with the smallest AUC value was NB (0.593) and DT (0.597) (figure 2).

Figure 2. Comparison of ROC curves for the eight ML models based on their respective AUC values. AUC, area under the curve; DT, Decision Trees; GBM, Gradient Boosting Machines; KNN, K-Nearest Neighbour; LR, Logistic Regression; ML, Machine Learning; NB, Naïve Bayes; RF, Random Forest; ROC, Receivers Operating Characteristics; SVM, Support Vector Machine; XGBM, eXtreme Gradient Boosting Model.

The performance of the final Xgboost model was evaluated on the holdout/test set. It demonstrated a high discriminative ability with an AUC value of 0.841 (figure 3A). The variable importance plot of the Xgboost model indicated that the single most important predictor (cluster 1) of measles vaccination dropout was Maternal age. This feature has a relative feature importance value of 0.148, being the single cluster. The rest of the features were considered as cluster 2 due to their low relative importance. Among those features, the husband decides on the healthcare need with 0.018 relative importance, followed by the mother who does not have an occupation with 0.016 value, the mother’s occupation of agriculture and domestic with 0.018, mother who has media exposure with 0.019 value and husband do not have any education with feature importance value of 0.021 (figure 3B). This feature importance, however, reveals only the relative importance of features and not their actual relationships with the predicted outcome (ie, measles vaccination status). To do this, we applied an independent model interpretability method called SHAP values.

Figure 3. (A) The performance of the Xgboost model on the test set. (B) The feature importance plot of the final Xgboost model. AUC, area under the curve; ROC, Receivers Operating Characteristics; XGBM, eXtreme Gradient Boosting Model.

Model interpretability

SHAP value is arguably the most powerful method for explaining how ML models make predictions. In this study, the Shapley Contribution and the Global Importance of the best model (ie, Xgboost) were both computed and plotted (figure 4A,B). These global interpretations were used to describe the expected behaviour of the ML model concerning the whole distribution of values for its input features. This was achieved by aggregating the SHAP values for individual instances across the entire population.

Figure 4. (A) Shapley Contribution Beeswarm plot of the Xgboost ML model. (B) Global Importance bar plot of the Xgboost ML model. ML, Machine Learning; SHAP, SHapley Additive exPlanations.

The Global Importance bar plot (figure 4B) examines the mean absolute SHAP value for each feature across all of the data. This plot quantifies, on average, the positive or negative magnitude of each feature’s contribution towards the predicted measles vaccination status. Features with higher mean absolute SHAP values are more influential. The input features are ranked from top to bottom by their mean absolute values for the entire data set. The mean absolute SHAP values show, on average, how much each feature impacts the predicted measles vaccination status, in the positive or negative direction. According to the plot, we see that the mother’s age is the most influential variable, contributing on average±0.155 value to each predicted measles vaccination status. Jehovah/Adventist religion is the second most influential feature, contributing on average±0.110 value, followed by Benishangul Gumuz region, a husband with no education, and mother with no occupation with average SHAP values of ±0.089, ± 0.078 and ±0.064, respectively. By contrast, the least informative feature, the Tigray region, contributes only ±0.0259.

Shapley Contribution Beeswarm plot (figure 4A) is a more complex and information-rich display of SHAP values that reveal not just the relative importance of features, but their actual relationships with the measles vaccination status. In this plot, for each variable, every instance of the data set appears as its point. The points are distributed horizontally along the X-axis according to their SHAP value. In places where there is a high density of SHAP values, the points are stacked vertically. Examining how the SHAP values are distributed reveals how a feature may influence the XGB model’s predictions. The colour bar corresponds to the raw values (not the SHAP values) of the features for each instance on the graph. If the value of a feature for a particular instance is relatively high, it appears as a golden dot. Relatively low feature values appear as purple dots. Examining the colour distribution horizontally along the X-axis for each variable provides insights into the general relationship between a feature’s raw value and its SHAP value. Because of those facts, figure 4A can examine how the underlying values (raw) of each feature relate to the model’s predictions. Accordingly, our XGB model predicted that small mother age, religion-Jehovah/Adventist, husband with no education, mother with no occupation, residents of the Oromia region, mother with primary education, large family size,⁶,¹⁰ residents of the Somali region and husband age greater than 61 have a positive impact (were predicted to be risk factors) on the model’s predicted output (ie, measles vaccination dropout).

Although the Xgboost model was the overall top performer in this analysis, the RF model was also considered due to its exceptional recall performance, achieving a perfect score of 1. To leverage the RF model’s robust capability in accurately detecting children who would drop out, a SHAP Beeswarm and bar plots were computed for this model. This approach provided additional insights into feature importance and their influence on the prediction of dropout of Measles one vaccination. This plot revealed that maternal age remained the most influential feature, exhibiting high SHAP values similar to those observed in the Xgboost model. However, the ranking of other features differed between the two models, highlighting variations in how they prioritise different predictors. This variation underscores the importance of evaluating multiple models to capture a comprehensive understanding of feature influence. By comparing these models, we gain deeper insights into the factors driving the target class detection (figure 5A,B). In contrast, the SHAP beeswarm and bar plots of the SVM model exhibited that the partner/father who decides on the healthcare needs of the family alone is the most influential predictor of measles vaccination dropout. Religion Adventist & Jehovah and Oromia region are the second and third most influential predictors (online supplemental figure 4).

Figure 5. (A) Shapley Contribution Beeswarm plot of the RF ML model. (B) Global importance bar plot of the RF ML model. ML, Machine Learning; RF, Random Forest.

Furthermore, using the SHAP additive property, the distribution of total SHAP values for the ‘region’ feature was plotted using a histogram and box plot. The histogram shows that the majority of data points are clustered at a SHAP value of 0, while there is some variation from 0 for certain bars. For those data points where the SHAP value deviates from 0, the ‘region’ feature influences the prediction, indicating that its impact may vary widely across different data points and regions. To observe the distribution and variability of SHAP values across different categories of the ‘region’ feature while minimising overplotting, a jitter plot was used. This plot displayed individual SHAP values for each region, with points jittered to avoid overlap. It reveals the spread and density of SHAP values, highlighting individual data points and any clustering or gaps (online supplemental figure 5).

Discussion

The dropout rate measures the proportion of children who begin but do not complete their vaccination schedule. It is intentionally used to evaluate the performance of immunisation services. A measles dropout rate exceeding 10% indicates a utilisation problem, meaning many children are not taking advantage of the available services. This secondary analysis revealed that approximately 39.9% of children had dropped out of measles vaccination, which is significantly higher than the 10% threshold recommended by the WHO.

This study compared eight ML algorithms to predict measles vaccination dropout among children in Ethiopia. According to the comparison, the XGBoost (XGB) algorithm achieved the highest predictive accuracy. Although ML models are often considered black boxes and can be difficult to interpret, significant advancements have been made in model interpretation over the last decade. This study utilised Shapley values, one of the most powerful methods for interpreting ML predictions.

While the XGBoost model demonstrated the highest overall performance, the RF model exhibited perfect recall ability. According to the Shapley plot for the best model, eXtreme Gradient Boosting model (XGBM), the following factors had a positive impact on the model’s output, indicating they were predicted as risk factors for measles vaccination dropout: mother’s age between 46 and 60 years, religion (Jehovah’s Witness/Adventist), a husband with no education, a mother with no occupation, residents of the Oromia region, a mother with primary education, large family size (6–10 members), residents of the Somali region and husband’s age greater than 61.

According to the SHAP plots of the best model (XGBoost), a mother’s age between 46 and 60 years is associated with negative SHAP values (points extending towards the left are increasingly golden), while lower maternal ages have positive SHAP values (points extending towards the right are increasingly purple). This suggests that children whose mothers are aged 46–60 are less likely to drop out of measles vaccination. Conversely, the SHAP plots of the RF model indicate that younger mothers, between 15 and 30 years old, are predictors of measles vaccination dropout (the dots extending towards the right are increasingly red). The importance of a mother’s age in predicting children’s vaccine dropout status has been documented in the literature and supports the findings of this study. Young mothers who marry early may become less attentive to vaccination schedules, often due to forgetfulness or a longer time gap between the third dose of the pentavalent vaccine and the measles vaccination, which can lead to missed vaccinations.^{11 31 32} This could also be related to the positive relationship between poorer vaccine-related knowledge among younger mothers and vaccination practices. Evidence suggests that higher knowledge about the importance of vaccinating children encourages and motivates mothers to ensure their child receives the vaccine.³³,³⁵ Since the variable knowledge of the mother regarding the importance of vaccination was not considered in this study, the relationship between vaccination status and the mother’s age could be confounded by this unexamined factor.

The SHAP values from the best-performing algorithm (XGBoost model) indicate that the second most important factor in measles vaccination dropout is adherence to the Adventist/Jehovah religion. Although studies on the association between religion and immunisation dropout are limited, religious beliefs have been reported to negatively affect vaccination uptake due to perceived spiritual principles against vaccination.^{36 37} Additionally, different religions around the world are associated with factors such as women’s empowerment, which is known to be an independent predictor of vaccination dropout status.^{38 39}

Husbands with no education and mothers with only primary education are also associated with a higher risk of dropping out of measles vaccination in our study. Several studies³⁷⁴⁰,⁴³ have shown that children from well-educated families are more likely to complete their vaccinations. Educated mothers and husbands generally have better access to information through media and health workers, which can enhance their health-seeking behaviour and lead to better decision-making regarding their family’s health and education.^{11 34} Additionally, Rammohan et al investigated the association between husbands’ education and measles vaccination status, finding that husbands’ educational status significantly influences measles vaccination uptake, independent of maternal education.⁴⁴

Mothers without employment were found to have a positive impact on children’s measles vaccination dropout. The occupational status of the mother is an important factor influencing vaccination dropout, as reported in numerous studies.^{12 39 45} Unemployed mothers often have less access to information and fewer opportunities to discuss vaccine-related concerns with educated individuals. This lack of awareness can hinder their understanding of their child’s health and healthcare needs, reducing their motivation to consistently take their children to vaccination centres.⁴⁶

Similarly, residing in the Oromia and Somali regions is associated with a higher risk of dropping out of measles vaccination. Both regions are major pastoral areas in Ethiopia.⁴⁷ Communities in pastoralist and semipastoralist regions often live in marginal, remote, conflict-prone and food-insecure areas. These regions typically face a shortage of health professionals and facilities, lack essential infrastructure such as transportation, electricity and safe water, and are situated at a greater distance from health facilities. These factors impact the availability of immunisation services and the health professionals providing them. Additionally, such conditions may negatively affect the knowledge, attitudes and practices of mothers and families regarding the importance of childhood vaccines.^{48 49}

Large family size is consistently associated with vaccination dropout status in various studies.⁵⁰,⁵² This relationship is attributed to several factors. A larger family size often increases the domestic workload, such as house chores and child-caring responsibilities, for mothers. This can lead to reduced media exposure, decreased health-seeking behaviour and a shortage of time to take the child to a vaccination centre as scheduled. Additionally, in larger families, parents may face greater challenges in organising and ensuring the quality of childcare. Limited resources, such as money for transportation, can further impact healthcare investments and access to vaccination services.⁵³

Paternal age greater than 61 is another significant factor impacting measles vaccination dropout status. Studies have shown that a father’s involvement is crucial for successful child immunisation.^{54 55} In Ethiopian society, cultural, traditional and religious norms often place the father as the head of the family, responsible for healthcare decisions. This can negatively affect a child’s immunisation status based on the father’s characteristics. Older fathers may have less knowledge about the importance of childhood vaccination, which can lead to hesitation and discouragement towards vaccinating their children.^{56 57}

This study utilised large-scale, nationally representative data with numerous features to enhance the prediction performance of the ML models. It implemented and compared eight ML models known for their superior classification performance in healthcare problems. Additionally, the study employed SHAP values, a powerful model interpretability tool, to explain the direction and strength of the best-performing ML model, the XGBM model. This approach addressed the model explainability gap found in previous similar studies.

In contrast, this study did not include ‘knowledge of the parents about measles and vaccination’ as a predictive feature. From an epidemiological perspective, this feature is known to be a significant determinant of vaccination practices, as reported in numerous studies. Therefore, its omission could potentially confound the relationship between several important features and the vaccination outcome.

Conclusion and recommendation

The measles dropout rate in the country was significantly higher than the WHO’s recommendation of less than 10%. To identify the best predictive model for measles vaccination dropout status, this study implemented and compared eight machine-learning algorithms. Among these, the XGBM was found to be the most effective for predicting measles vaccination dropout among 12–23 month-old children in Ethiopia. Model interpretability analysis using Shapley values revealed that factors, such as young maternal age, Jehovah/Adventist religion, husbands with no education, mothers with primary education, unemployed mothers, residents of the Oromia and Somali regions, large family size⁶,¹⁰ and husbands older than 61, were the most influential features positively affecting measles dropout status. Healthcare professionals and stakeholders should leverage this information to develop strategies aimed at reducing measles vaccination dropout rates. Potential interventions include launching awareness campaigns to improve public understanding of measles vaccination, providing targeted health education at public gatherings like churches and mosques and collaborating with religious institutions and health extension workers to maintain trust and credibility in the vaccines offered through EPI programmes. Additionally, implementing region-specific interventions, especially in pastoralist and semipastoralist areas, would help keep the measles vaccination dropout rate below acceptable levels.

Data availability statement

Data are available in a public, open access repository.