Abstract
Background
Accurate measurement of smoking in population-based administrative health data (AHD) poses challenges due to the indirect nature of smoking-related information collection. While most studies use rule-based algorithms (RBAs) based on diagnosis codes, model-based algorithms (MBAs) utilizing machine learning (ML) with diverse data features might have better sensitivity and accuracy. We developed ML model-based algorithms (MBAs) for ascertaining smoking in AHD and compared them to RBAs.
Methods
We conducted a retrospective cohort study using AHD (hospital abstracts, medical claims, and prescription drug records) from April 1, 2012, to March 31, 2020, from Manitoba, Canada. The study included adults (≥ 18 years) from a clinical registry containing self-reported current smoking. Clinical data were linked with up to five years of hospital records, physician billing claims, and prescription medication records. RBAs were based on diagnosis codes for tobacco use and nicotine dependence medication. MBAs, constructed using Random Forest (RF) and Least Absolute Shrinkage and Selection Operator (LASSO) models, included smoking indicators, comorbid condition, and sociodemographic factors. Training and test datasets were used to develop and evaluate the MBAs, respectively. Sensitivity, specificity, positive and negative predictive values (PPV, NPV), balanced accuracy, and their 95% confidence intervals (CIs) were estimated.
Results
The cohort comprised 24,718 individuals (88.6% female); prevalence of current smokers was 10.0%. A comprehensive RBA had sensitivity 23.3% (95% CI: 20.3–26.5), specificity 98.9% (95% CI: 98.7–99.2), and PPV 70.9% (95% CI: 65.1–76.1). An MBA based on RF had sensitivity 66.8% (95% CI: 63.3–70.2), specificity 77.8% (95% CI: 76.8–78.8), and PPV 25.1% (95% CI: 23.8–26.4). NPV was consistently above 90.0%. MBAs had higher balanced accuracy than RBAs. Stratified analyses by sex and residence location revealed differences in estimates for MBAs and the RBAs. The number of years of AHD did not affect the MBA results. While MBAs had better sensitivity, RBAs had better specificity.
Conclusions
Our study highlights the potential of comprehensive data integration and ML methods to improve the sensitivity and accuracy of smoking identification in AHD. Balancing accurate smoker identification with the risk of false positives is crucial when choosing an algorithm to ascertain current smokers using AHD.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12874-026-02839-8.
Keywords: Tobacco Use Disorder, Accuracy, Data Linkage, Sensitivity, Specificity, Algorithms, Machine Learning
Background
Administrative health data (AHD) are a rich source of information collected and maintained for the purpose of managing and supporting the healthcare system [1]. These data typically capture hospital visits, physician billing claims, and drug dispensation records. The wide coverage of these data sources provides opportunities to conduct population-based studies. Smoking is an important variable in population-based studies for understanding the burden of smoking-related diseases [2], designing effective public health interventions [3], evaluating policy interventions [4], and addressing health disparities [5]. However, because smoking is not directly recorded in AHD, obtaining valid information is challenging. Therefore, it is important to evaluate the validity of AHD for accurately measuring smoking behaviors.
Current approaches to identify smoking status in AHD often rely on rule-based algorithms (RBAs), which are deterministic case definitions that involve measuring the type and frequency of specific codes, such as diagnosis codes in the records of an individual over a specific timeframe [6]. However, RBAs for smoking ascertainment in electronic health data, including AHD, have low sensitivity [7, 8] and accuracy [9]. To illustrate, a study evaluating RBAs to identify tobacco use in US Medicare claims reported sensitivity as low as 9.0%, highlighting substantial under-detection of smokers [8]. Model-based methods offer an alternative for algorithm development [10] with the capacity to incorporate a diverse set of data elements, including demographics, diagnosis codes, billing codes, and prescription medications associated with smoking, and to assess their relative importance in case ascertainment. By leveraging a broader spectrum of information, machine-learning (ML) model-based methods can potentially improve the sensitivity and accuracy [11] of smoking ascertainment algorithms over their rule-based counterparts.
A recent systematic review that assessed the validity of smoking measurement in AHD and electronic medical records (EMR) found that the majority of the algorithms were rule-based and that model-based algorithms were primarily developed for EMR data [12]. Algorithms to identify smoking status in AHD are frequently validated using non-clinical data sources prone to misclassification errors, such as survey data [13], birth certificates [14], and medical chart reviews [15]. Moreover, constructing model-based algorithms (MBAs) for a single AHD source is challenging due to a limited number of relevant indicators for model development. However, the availability of multiple smoking indicators in different linkable AHD sources and its population-based nature suggest the viability of using ML models to construct smoking ascertainment algorithms in linked data.
This study aimed to assess the validity of AHD to ascertain smoking for a clinical registry-based cohort using multiple linked data sources. The objectives were to (1) develop ML MBAs to identify current smoking status and (2) investigate whether the MBAs result in higher validity estimates than RBAs.
Methods
We conducted a retrospective cohort study with three linked AHD data sources from April 1, 2012, to March 31, 2020. We followed the Reporting of Studies Conducted Using Observational Routinely-collected Data guidelines [16] (Additional file 1).
Data sources
Study data were from the Population Research Data Repository housed at the Manitoba Centre for Health Policy (MCHP) in Manitoba, Canada [17]. The province of Manitoba has a universal healthcare system, so almost all contacts with the healthcare system are captured in AHD. Study data sources included the Manitoba health insurance registry, hospital abstracts, medical (i.e., physician billing) claims, prescription medication dispensations, and the Statistics Canada Census. These data were used to develop algorithms using rule-based and ML methods. The Manitoba Health Insurance Registry contains individual-level demographic and residential information for everyone registered to receive health services in Manitoba. Hospital abstracts encompass records from hospital stays, while medical claims include documentation of physician visits in both offices and outpatient departments. Diagnoses in hospital abstracts and physician billing claims are coded using the International Classification of Diseases (ICD) codes. Specifically, hospital abstracts use the 10th version with Canadian adaptations (ICD-10-CA) and physician billing claims use the 9th version with clinical modifications (ICD-9-CM). Prescription medication dispensations from community-based pharmacies are captured in the Drug Program and Information Network (DPIN) database. The Anatomical Therapeutic Chemical (ATC) Classification System [18] codes for classifying pharmaceutical substances are used to capture information in DPIN. The Canada Census [19] is a nationwide population-based data source that includes aggregated socio-demographic information that can be used to construct income quintiles based on postal codes in the Manitoba Health Insurance Registry.
The Manitoba bone mineral density (BMD) clinical registry [20] was used as the validation data source, considering the availability of information on smoking status and its completeness and accuracy. A BMD test [21] is a diagnostic technique employed to quantify the mineral density, particularly calcium, within bone structures. These tests play a pivotal role in identifying individuals vulnerable to fractures resulting from diminished bone density and are crucial for the timely detection and treatment of osteoporosis. These tests are usually advised for postmenopausal women, older individuals, those with specific medical conditions or medications impacting bone health, and individuals with a history of fractures. The BMD registry contains BMD tests performed within Manitoba; it includes men and women, many of whom are 50 years of age and older. The data on smoking status documented in this registry was obtained directly from patients during clinical visits, ensuring high accuracy when compared to survey responses that are prone to self-report bias. This registry’s self-reported smoking status information is frequently used as a covariate in in fracture risk prediction models [22].
Study cohort
The study cohort included adults (age ≥ 18 years) captured in the BMD registry with a service date between April 1, 2017, and March 31, 2020, and linkable to administrative data through a scrambled personal health identification number (PHIN) at the individual level. The PHIN is a unique nine-digit number given by Manitoba Health to individuals registered for health insurance in Manitoba to receive treatment at facilities submitting electronic claims [23]. If there was more than one record for a cohort member, then a single record was selected randomly to ascertain smoking. The consistency of current smoking status across multiple records for a cohort member was also assessed.
Target measure
The target measure (dependent variable in models) was current smoking status (1 = Yes, 0 = No) from the BMD registry. The registry data do not capture information on whether an individual had previously smoked or the amount that an individual currently smokes. The service date (i.e., date of appointment for BMD test) in the BMD registry was the index date.
Indicators of smoking status in administrative health data
Independent variable identification used up to five years of AHD before each individual’s index date. This inclusion criterion (Fig. 1) was used to ensure that every individual in the validation cohort was followed for the same length of time during the study period. Information from the AHD sources included diagnosis codes for smoking-related conditions, such as a history of tobacco use. The use of prescribed medications such as nicotine dependence drugs was measured from the DPIN data. The Manitoba Health Insurance Registry was used to obtain age (in years), biological sex, and place of residence (urban, rural). Income quintiles, measured at the level of a dissemination area (consisting of 400–700 individuals) based on total household income, were obtained from the Statistics Canada Census [24]. Indicators from administrative data were used as independent variables in models for constructing current smoking status identification algorithms (i.e., variables related to nicotine dependence, chronic obstructive pulmonary disease [COPD], asthma, and smoking cessation drugs; biological sex, place of residence, age, and income quintiles). The descriptions of the indicators is provided in Additional file 2.
Fig. 1.
Cohort flow diagram
Rule-based algorithm (RBA)
Two rule-based algorithms were developed using the diagnosis codes from AHD. The distinction between these two algorithms lies in ATC code for Bupropion, which is a smoking cessation aid also used to treat depression.
RBA1
One or more hospital separation records with at least one of the following diagnosis codes [25]: ICD-10-CA code F17.2 (nicotine dependence), ICD-10-CA code Z72.0 (problems related to lifestyle: tobacco use) OR one or more physician claims with at least one of the following diagnosis codes: ICD-9-CM code 305.1(nicotine dependence), ICD-9-CM code V15.82 (personal history of tobacco use), OR one or more pharmaceutical dispensation records with at least one of the following ATC drug codes [26]: N07BA03 (Varenicline), N07BA01 (Nicotine).
RBA2
The ATC drug code [27]: N06AX12 (Bupropion), was added to RBA1.
Statistical analysis
Cohort characteristics were reported using frequencies and percentages or mean and standard deviation stratified by the individuals’ current smoking status. Descriptive analysis was performed in SAS® version 9.4 (SAS Institute, Cary, North Carolina, United States). Model-based analyses were conducted using R [28]. We conducted analyses separately for one to five years of administrative data preceding the index date to facilitate comparisons across years since previously validated algorithms used different lengths of time preceding the index date to identify smoking status [25–27]. MBAs were constructed using ML models developed with and without sociodemographic variables: age, biological sex, place of residence, and income quintile. The missing (i.e., Not Found) category for place of residence was combined with the rural category and for income quintile the missing category was combined with the highest income quintile category (Q5) to increase sample sizes in these categories for model fitting.
Model-based algorithm (MBA)
We used two ML models, random forest (RF) and least absolute shrinkage and selection operator (LASSO) logistic regression to predict current smoking status. We selected these two models for comparison since model performance has been shown to vary with the characteristics of the dataset, the magnitude of imbalance in the target variable, and the specific task or domain in which they are applied [29, 30]. The ability to avoid overfitting and identify important variables also motivated the selection of these two ML models. The RF is an ensemble learning method that operates by constructing a large number of decision trees, where each tree is built by recursively partitioning the data into nodes and making splits based on specific criteria, such as the minimum number of observations in a node. The results from individual trees are then aggregated with a majority vote. RF overcomes the main limitations of single decision trees by reducing the variance and decorrelating the trees with bootstrapping and aggregating, respectively [31]. LASSO extends logistic regression by favoring a sparse model, setting less impactful variable coefficients to zero through a penalty term proportional to the coefficients [32].
The study cohort was randomly stratified into a training set (70% of the cohort) and a test set (30%), ensuring that the prevalence of current smoking was consistent in both sets. An optimization set (30%) from the training data was randomly chosen to find the optimal value for hyperparameters using the grid search approach [33]. The hyperparameters for the RF model were the number of variables randomly sampled as candidates at each split and the minimum number of observations required to split an internal node during the construction of a decision tree for the model. The number of trees was set at 1500 for each RF model [34]. The single hyperparameter for LASSO logistic regression was the multiplier of the penalty term, known as lambda (the higher the value of lambda, the stronger the penalty and the more coefficients are set to zero). The hyperparameter estimates that resulted in the highest area under the curve (AUC) of the receiver operating characteristic (ROC) curve were chosen to train the model. The relative importance of variables was measured with the contribution to the reduction in impurity and the magnitude of coefficients, respectively, for RF and LASSO logistic regression models using the VarImp function from the caret R package [35].
The trained models were applied to the test dataset to estimate the predicted probability of current smoking. When choosing the classification threshold from predicted probabilities, we aimed for a balance between accurately identifying both positive and negative cases by giving equal importance to sensitivity and specificity. Hence, the threshold probabilities corresponding to the highest Youden’s J index from ROC curve analyses were chosen for an individual to be classified as a current smoker using the MBAs [36].
Performance evaluation
For each MBA and RBA, performance was assessed using the test set based on validity measure estimates. The validity measures were sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and balanced accuracy [37]. Sensitivity estimates the proportion of true smokers correctly identified by an algorithm, while specificity estimates the proportion of true non-smokers accurately identified as non-smokers. PPV estimates the probability that an individual classified by an algorithm as a smoker is a true smoker. In contrast, NPV estimates the probability that an individual identified as a non-smoker is a true non-smoker. Balanced accuracy is a summary measure that integrates sensitivity and specificity to estimate an algorithm’s classification performance. By equally weighting the correct identification of smokers and non-smokers, it offers a balanced evaluation of accuracy across classes (smokers and non-smokers). However, the relevance of balanced accuracy may vary depending on the study context, as some scenarios may prioritize one measure over the other, such as focusing on high specificity when reducing false positives is crucial. For each algorithm, we reported both overall and stratified estimates of validity measures by biological sex and location of residence. The corresponding 95% confidence interval (CI) was also reported. DeLong’s non-parametric test was used to compare estimates for MBAs [38] using a nominal α = 0.05.
Results
We identified 25,712 individuals in the BMD registry, of which 24,718 (96.1%) met the inclusion criteria (Fig. 1). Only 465 (1.9%) cohort members had more than one record in the registry, for whom a single record was randomly selected. For the vast majority (97.0%) of those having more than one registry record, current smoking status was consistent across all records.
Cohort characteristics
The prevalence of current smokers within the study cohort was 10.0% (Table 1). The majority of cohort members were female (88.6%), residing in urban areas (70.2%), and older adults (average age = 67.0 years). Significant differences were observed in the percentages of current smokers and non-smokers by biological sex, residence location, and income quintile. Notably, within the lowest income quintile, the prevalence of current smokers exceeded that of non-smokers (26.4% vs. 18.4%). In contrast, within the highest income quintile, the percentages of current smokers and non-smokers were 12.5% and 22.2%, respectively. On average, current non-smokers tended to be older than smokers. Additionally, the prevalence of current smokers exceeded that of non-smokers across various smoking-related indicators, including diagnoses of tobacco use, smoking-related comorbid conditions, and prescription medication dispensations. For instance, with five years of AHD usage, the proportion of current smokers was 22.1 times that of non-smokers for the diagnosis of nicotine dependence in physician claims.
Table 1.
Characteristics of the study cohort stratified by self-reported current smoking status (N = 24,718)
| Characteristic | Current smoker n = 2475 (10.0%) |
Non-smoker n = 22,243 (90.0%) |
|---|---|---|
| Sex | ||
| Male | 360 (14.5) | 2452 (11.0) |
| Female | 2115 (85.5) | 19,791 (89.0) |
| Residence location at index date | ||
| Urban | 1541 (62.3) | 15,813 (71.1) |
| Rural | 905 (36.5) | 6257 (28.1) |
| Not found | 29 (1.2) | 173 (0.8) |
| Income quintile | ||
| Q5 (highest) | 309 (12.5) | 4942 (22.2) |
| Q4 | 415 (16.7) | 4506 (20.3) |
| Q3 | 507 (20.5) | 4596 (20.6) |
| Q2 | 562 (22.7) | 3942 (17.7) |
| Q1 (lowest) | 653 (26.4) | 4084 (18.4) |
| Not found | 29 (1.2) | 173 (0.8) |
| Age at index date, mean (SD) | 62.4 (10.3) | 67.6 (10.9) |
| Diagnoses | ||
| Nicotine dependence or tobacco use problems related to lifestyle in hospital abstracts / personal history of tobacco use in medical claims | 30 (1.2) | 24 (0.11) |
| Nicotine dependence diagnosis in medical claims | 258 (10.4) | 105 (0.47) |
| COPD /asthma diagnosis in hospital abstracts | 140 (5.6) | 484 (2.2) |
| COPD in medical claims | 675 (27.3) | 1718 (7.7) |
| Asthma in medical claims | 460 (18.6) | 2949 (13.3) |
| Use of respiratory system treatment drug (Salbutamol) in prescription drug records | 598 (24.3) | 3214 (14.5) |
| Use of medication primarily recommended as an antidepressant and for smoking cessation in prescription drug records (Bupropion) | 159 (6.4) | 552 (2.5) |
| Use of medication to quit smoking in prescription drug records (Varenicline) | 276 (11.2) | 186 (0.84) |
| Use of medication to aid in smoking cessation in prescription drug records (Nicotine) | 123 (4.9) | 59 (0.27) |
Diagnoses are based on five years of administrative health data; Standard deviation (SD); Chronic obstructive pulmonary disease (COPD); Quintile (Q); Values reported are frequency (%) unless otherwise stated
Overall performance of algorithms
For the MBAs, the AUC estimates for the RF models were slightly higher than the AUC estimates for the LASSO models (Fig. 2). The AUC for the RF models ranged from 0.74 to 0.79, while for the LASSO models the range was 0.72 to 0.77 for different numbers of years of AHD for independent variable identification. The comparisons of AUC estimates for the algorithms (Additional file 3) using DeLong’s test indicated statistically significant differences between the pair of MBAs except for the LASSO model versus the RF model using five years of AHD.
Fig. 2.
Comparison of model-based algorithms (independent variables identified based on five years of administrative health data) for area under the receiver operating characteristic curve in test data. Random forest (RF); Least Absolute Shrinkage and Selection Operator (LASSO)
The hyperparameter estimates for RF models did not vary substantially across the number of years of AHD for independent variable identification (number of variables randomly sampled as candidates at each split: 2, 3; minimum number of observations required to split: 31, 35). For the LASSO models, the estimates of lambda ranged between 0.0004 and 0.0046. The estimated optimized threshold for the predicted probabilities varied from 0.088 to 0.1146.
Figure 3 depicts the ranking of the importance of variables resulting from the RF and LASSO models. Age, diagnosis of nicotine dependence in medical claims, and diagnosis of chronic obstructive pulmonary disease in medical claims consistently appeared in this list of variables. The income quintiles also contributed notably to smoking ascertainment in AHD using MBAs.
Fig. 3.
Top ten most important variables for Random Forest (RF) and Least Absolute Shrinkage and Selection Operator (LASSO) models using independent variables identified based on five years of administrative health data
Estimates of sensitivity, NPV, and balanced accuracy estimates for MBAs were higher than the estimates for RBAs for all years (Table 2). RBAs performed better than MBAs for specificity and PPV estimates. MBAs based on models excluding sociodemographic variables had low sensitivity and high specificity.
Table 2.
Random forest (RF), Least Absolute Shrinkage and Selection Operator (LASSO) logistic regression, and rule-based algorithm (RBA) performance for test data (n = 7414) by number of years of administrative health data used for identifying independent variables
| Validity measure | Algorithm | Number of years of administrative data before cohort index date | ||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| Sensitivity | RBA1 |
7.5 (5.7, 9.7) |
11.7 (9.5, 14.3) |
15.6 (13.1, 18.4) |
20.3 (17.5, 23.4) |
23.3 (20.3, 26.5) |
| RBA2 |
9.7 (7.7, 12.1) |
14.7 (12.2, 17.4) |
19.3 (16.5, 22.3) |
23.9 (20.9, 27.2) |
27.3 (24.2, 30.7) |
|
| RF |
63.6 (60.0, 67.1) |
66.8 (63.3, 70.2) |
62.5 (58.9, 66.0) |
64.3 (60.7, 67.7) |
66.8 (63.3, 70.2) |
|
| LASSO |
61.7 (58.1, 65.2) |
70.1 (66.6, 73.4) |
65.6 (62.1, 69.0) |
65.8 (62.2, 69.2) |
68.5 (64.9, 71.8) |
|
| RF* |
0.7 (0.0, 1.2) |
0.4 (0.0, 0.8) |
0.6 (0.0, 1.0) |
0.4 (0.0, 0.8) |
0.9 (0.2, 1.6) |
|
| LASSO* |
1.2 (0.4, 2.0) |
0.1 (0.0, 0.5) |
0.3 (0.0, 0.7) |
0.4 (0.0, 0.8) |
0.4 (0.0, 0.8) |
|
| Specificity | RBA1 |
99.8 (99.6, 99.9) |
99.6 (99.5, 99.8) |
99.4 (99.2, 99.6) |
99.3 (99.0, 99.5) |
98.9 (98.7, 99.2) |
| RBA2 |
98.6 (98.3, 98.9) |
98.1 (97.8, 98.4) |
97.6 (97.2, 98.0) |
97.2 (96.8, 97.6) |
96.6 (96.1, 97.0) |
|
| RF |
74.4 (73.3, 75.4) |
72.3 (71.2, 73.4) |
78.5 (77.5, 79.5) |
78.4 (77.3, 79.3) |
77.8 (76.8, 78.8) |
|
| LASSO |
73.3 (72.3, 74.4) |
66.7 (65.5, 67.8) |
72.9 (71.8, 74.0) |
74.8 (73.7, 75.8) |
73.7 (73.6, 74.8) |
|
| RF* |
99.6 (99.5, 99.8) |
99.6 (99.5, 99.8) |
99.5 (99.3, 99.7) |
99.8 (99.7, 99.9) |
99.3 (99.1, 99.5) |
|
| LASSO* |
99.1 (98.9, 99.3) |
99.6 (99.5, 99.7) |
99.7 (99.6, 99.8) |
99.8 (99.7, 99.9) |
99.8 (99.7, 99.9) |
|
| PPV | RBA1 |
78.9 (67.9, 86.7) |
78.3 (69.9, 85.0) |
74.3 (67.1, 80.5) |
75.9 (69.6, 81.2) |
70.9 (65.1, 76.1) |
| RBA2 |
44.1 (36.9, 51.6) |
46.6 (40.5, 52.7) |
47.3 (42.1, 52.6) |
48.7 (44.0, 53.5) |
47.2 (42.9, 51.5) |
|
| RF |
21.7 (20.5, 22.8) |
21.2 (20.1, 22.2) |
24.4 (23.1, 25.8) |
24.8 (23.5, 26.1) |
25.1 (23.8, 26.4) |
|
| LASSO |
20.5 (19.4, 21.6) |
18.9 (18.1, 19.8) |
21.2 (20.1, 22.3) |
22.5 (21.4, 23.7) |
22.4 (21.4, 23.6) |
|
| RF* |
17.8 (3.7, 32.0) |
10.3 (0.0, 21.4) |
13.5 (2.5, 24.5) |
27.3 (0.1, 53.6) |
13.7 (4.2, 23.2) |
|
| LASSO* |
13.2 (5.2, 21.3) |
20.0 (0.0, 55.5) |
20.0 (0, 44.7) |
23.1 (0.2, 45.9) |
20.0 (0.0, 40.0) |
|
| NPV | RBA1 |
90.6 (90.5, 90.8) |
91.0 (90.8, 91.2) |
91.3 (91.1, 91.6) |
91.8 (91.5, 92.1) |
92.1 (91.8, 92.3) |
| RBA2 |
90.7 (90.6, 91.0) |
91.2 (90.9, 91.4) |
91.5 (91.3, 91.8) |
91.9 (91.7, 92.3) |
92.3 (92.0, 92.6) |
|
| RF |
94.8 (94.3, 95.3) |
95.1 (94.7, 95.6) |
94.9 (94.5, 95.4) |
95.2 (94.7, 95.6) |
95.5 (95.0, 95.9) |
|
| LASSO |
94.5 (94.0, 95.0) |
95.2 (94.7, 95.7) |
95.0 (94.5, 95.5) |
95.2 (94.7, 95.6) |
95.5 (95.0, 95.9) |
|
| RF* |
90.0 (89.3, 90.7) |
89.9 (89.3, 90.7) |
90.0 (89.3, 90.7) |
90.0 (89.3, 90.7) |
90.0 (89.3, 90.7) |
|
| LASSO* |
90.0 (89.3, 90.7) |
89.9 (89.3, 90.7) |
90.0 (89.3, 90.7) |
90.0 (89.3, 90.7) |
90.0 (89.3, 90.7) |
|
| Balanced accuracy | RBA1 |
53.7 (52.5, 54.8) |
55.7 (54.6, 56.8) |
57.5 (56.3, 58.6) |
59.8 (58.6, 60.9) |
61.1 (59.9, 62.2) |
| RBA2 |
54.2 (53.1, 55.3) |
56.4 (55.3, 57.9) |
58.4 (57.3, 59.5) |
60.6 (59.5, 61.7) |
61.9 (59.9, 62.1) |
|
| RF |
68.9 (67.8, 69.9) |
69.6 (68.5, 70.6) |
70.5 (69.5, 71.5) |
71.3 (70.3, 72.3) |
72.3 (71.3, 73.3) |
|
| LASSO |
67.5 (66.4, 68.6) |
68.4 (67.3, 69.4) |
69.3 (68.3, 70.3) |
70.3 (69.3, 71.3) |
71.1 (70.1, 72.1) |
|
| RF* |
50.2 (49.8, 51.1) |
50.0 (49.4, 50.4) |
50.0 (49.4, 50.4) |
50.1 (49.7, 50.6) |
50.1 (49.7, 50.6) |
|
| LASSO* |
50.1 (49.7, 50.6) |
50.0 (49.4, 50.4) |
50.1 (49.7, 50.6) |
50.1 (49.7, 50.6) |
50.1 (49.7, 50.6) |
|
* Model excluded sociodemographic variables of age, biological sex, income quintile, and place of residence; Boldface font denotes the highest estimates for each validity measure; Positive predictive value (PPV); Negative predictive value (NPV)
Overall, the estimates for the MBAs using the RF model and LASSO model were similar. The number of years of AHD increased the sensitivity estimates for RBAs considerably. Estimates of sensitivity, NPV, and balanced accuracy were slightly higher for RBA2 than for RBA1. Sensitivity estimates ranged from 7.5% (95% CI: 5.7–9.7) to 68.5% (95% CI: 63.3–70.2), with the highest and lowest values derived from the LASSO model (five years of AHD) and RBA1 (one year of AHD), respectively. Specificity estimates varied between 66.7% (95% CI: 65.5–67.8) and 99.8% (95% CI: 99.6–99.9), with RBA1 (one year of AHD) and LASSO (two years of AHD) yielding the highest and lowest values, respectively. PPV estimates ranged from 18.9% (95% CI: 18.1–19.4) to 78.9% (95% CI: 67.9–86.7), with the RBA1 (one year of AHD) and the LASSO model (two years of AHD) providing the highest and lowest values, respectively. For all algorithms, NPV estimates remained consistent across the years of AHD usage.
Performance of algorithms by demographic characteristics
The MBAs showed substantial differences between females and males (Additional file 4) for sensitivity and specificity estimates. PPV estimates differed for females and males for RBA1. The magnitude of the differences decreased with the number of years of AHD. For the RF model based on one year of AHD, estimates of sensitivity for females and males were 59.9% (95% CI: 56.0-63.8) and 85.2% (95% CI: 77.1–91.3), respectively. For females, the specificity estimate for the LASSO model based on one year of AHD was 75.5% (95% CI: 74.4–76.6), and it was 56.2% (95% CI: 52.5–69.8) for males. The RBA1 based on one year of AHD produced PPV estimates of 80.5% (95% CI: 68.9–88.5) and 66.7 (33.7, 88.7), respectively, for females and males.
Sensitivity and specificity estimates for the MBAs also differed based on place of residence (Additional file 5). Specifically, sensitivity estimates were higher for rural/not-found places of residence than for urban residents, whereas specificity was relatively higher for urban residents than for residents of non-urban areas. For example, sensitivity estimates from the LASSO model based on five years of AHD for urban and rural/not found places of residence were 62.2% (95% CI: 55.7–67.6) and 78.8% (95% CI: 71.7, 83.4) respectively. For the same algorithm, specificity estimates were 80.1% (95% CI: 76.8, 83.9) and 57.1% (95% CI: 54.5, 60.9), respectively, for urban and rural/not found places of residence.
Discussion
Smoking is a risk factor of interest in many disease risk prediction models developed using AHD [39, 40]. Valid identification of smoking status may improve the accuracy of risk prediction models. Consequently, it is imperative to conduct algorithm validation studies to assess the accuracy of smoking ascertainment in AHD [41]. While RBAs for smoking status identification in AHD have been subject to numerous validation studies [12], the validation of MBAs has been limited. Moreover, there is a gap in the literature about using multiple AHD sources to construct model-based smoking ascertainment algorithms and using reliable validation data sources to validate these algorithms. Furthermore, smoking algorithm validation studies have not often compared MBAs with rule-based counterparts in the same cohort. This study used ML model-based methods to develop algorithms for smoking ascertainment in AHD, validated them with a clinical data source, and compared MBAs against RBAs.
Performance of RBAs
In line with previous research [7], we found low sensitivity and high specificity for RBAs. Our study showed that including a less specific diagnosis code in RBA2 improved sensitivity but reduced PPV compared to RBA1, in contrast with previous research on ever-smoking ascertainment [27]. Additionally, we noted that the sensitivity estimates of RBAs increased with greater numbers of years of AHD, while specificity remained consistent, aligning with findings from a previous study [25]. While PPV estimates dropped with an increase in the number of years of AHD in the aforementioned study, our PPV estimates remained stable.
Performance of MBAs
The specificity estimates for MBAs surpassed their sensitivity estimates, aligning with previous validation studies of MBA-based ascertainment of current smoking [42] and ever-smoking [43] in claims data. Notably, in our study, the difference between sensitivity and specificity estimates was narrower, with relatively higher sensitivity estimates compared to prior investigations, possibly due to our use of Youden’s J index for cut-off probability selection. Our MBAs had balanced accuracy and AUC estimates similar to those from earlier studies. However, our PPV estimates were lower than those of an earlier study on current smoking [42]. The important variables (e.g., age and drugs used for addictive disorders) identified by our MBAs mirrored those identified in an ever-smoking ascertainment study. The validity measures were comparable between algorithms based on RF and LASSO models in our study, which differs from a previous study on predicting nicotine addiction (e-cigarette and waterpipe) [44]. In that study, RF models showed better performance compared to LASSO models when using the National Youth Tobacco Survey data from the US. These findings emphasize the necessity of refining MBAs to enhance the accuracy of smoking ascertainment algorithms.
Comparison between RBAs and MBAs
Our findings demonstrated that MBAs generally outperformed RBAs in identifying smokers. Specifically, higher sensitivity and balanced accuracy estimates were observed for MBAs compared to RBAs. Despite this better performance of MBAs, RBAs demonstrated higher specificity and PPV. These findings contrast with those of a validation study on rheumatoid arthritis, where MBA and RBA exhibited similar validity measure estimates [45]. The difference between MBAs and RBAs on the estimates of validity measures is likely attributed to the information incorporated into their construction. In particular, RBAs included diagnosis codes that are potential indicators of smoking [25–27], while MBAs also incorporated sociodemographic characteristics and comorbid conditions in addition to these indicators.
Impact of number of years of administrative data and sociodemographic factors
We observed variation in sensitivity estimates across different numbers of years of AHD, highlighting the importance of considering this attribute of algorithm development. Sensitivity and specificity estimates varied by biological sex and place of residence, particularly for MBAs. The ML models identified age and income as important variables contributing to MBAs. While specificity estimates improved with sociodemographic variable exclusion from the MBAs, sensitivity estimates dropped. These results underscore the necessity of accounting for sociodemographic factors in algorithm refinement and validation [46].
Study implications
The findings have a number of implications for further research and public health. Developing MBAs utilizing multiple AHD sources represents a scalable and replicable methodology for enhancing the accuracy of smoking ascertainment in population-based studies. Moreover, incorporating sociodemographic variables into MBAs can improve their capacity to identify disparities in smoking-related behaviors across different population subgroups, providing critical insights for the design and implementation of targeted public health interventions.
Strengths and limitations
The strengths of this study include the use of a reliable clinical registry data source for algorithm validation, which has been repeatedly used to develop risk prediction models for osteoporosis-related fractures [47, 48]. Moreover, we compared the performance of RBAs and MBAs for smoking ascertainment in the same cohort. Furthermore, we used three population-based AHD sources for algorithm development.
Some limitations should be considered when interpreting our findings. Our study focused exclusively on a single registry-based cohort, predominantly comprised of elderly females with specific healthcare needs. This may limit the generalizability of our findings across the entire population. Second, the ML models were trained on an imbalanced dataset, potentially introducing bias towards the majority class [49]. To circumvent potential inaccuracies stemming from this imbalance, we focused on additional metrics beyond accuracy to evaluate the validity of the algorithms. Third, we selected a single threshold to evaluated performance of the MBAs, which maximized both sensitivity and specificity. Selecting a different threshold would have affected our comparative assessment of the relative performance of the MBAs and RBAs, but would not have affected our conclusion that MBAs exhibited better sensitivity than RBAs. Finally, the algorithms only underwent internal validation, which could affect the generalizability of the estimates of algorithm validity measures in other populations.
Future directions
Further validation in external datasets is essential to confirm the robustness and reliability of the MBAs. Alternatively, artificially generated synthetic data that have the statistical properties of real-world validation data might be used to produce augmented training data to more accurately evaluate the performance of MBAs. Previous studies have demonstrated the use of synthetic data to address class imbalance, preserve privacy, and enhance model training. For example, studies have used synthetic data to create balanced datasets that better reflect the population characteristics [50, 51]. Subsequent research could also leverage the use of longitudinal data to uncover subtle shifts in smoking behavior. For example, a longitudinal model [52] might identify individuals who have recently quit smoking or those who have gradually increased their smoking over time, providing insights into evolving patterns of tobacco use. Including additional data sources that capture variables relevant to smoking, such as hospital outpatient data, could be another way to improve the performance of the developed algorithms.
Conclusions
By leveraging analytic techniques and incorporating a broad set of data elements, our study demonstrates a promising avenue for improving the accuracy of smoker identification in AHD for population-based studies. However, RBAs achieved high specificity and PPV, highlighting their importance for identifying smoking status. Hence, we suggest adopting a nuanced approach. Rather than exclusively relying on model-based or rule-based methods, we recommend assessing the context based on the specific research questions to determine the most suitable method.
Supplementary Information
Acknowledgements
The authors acknowledge the Manitoba Centre for Health Policy for using data contained in the Manitoba Population Health Research Data Repository (the Provincial Health Research Privacy Committee # P2023-29). The presented study findings and interpretations reflect the authors’ perspectives. No formal endorsement by the Manitoba Centre for Health Policy, Manitoba Health, or any other data providers is implied or should be inferred.
Abbreviations
- AHD
Administrative health data
- ML
Machine learning
- EMR
Electronic medical records
- MCHP
Manitoba Centre for Health Policy
- ICD
International Statistical Classification of Diseases and Related Health Problems
- ATC
Anatomical Therapeutic Chemical
- DPIN
Drug Program and Information Network
- BMD
Bone mineral density
- PHIN
Personal health identification number
- RBA
Rule-based algorithm
- MBA
Model-based algorithm
- RF
Random forest
- LASSO
Least Absolute Shrinkage and Selection Operator
- AUC
Area under the curve
- ROC
Receiver operating characteristic
- CI
Confidence interval
- PPV
Positive predictive value
- NPV
Negative predictive value
- COPD
Chronic obstructive pulmonary disease
Authors’ contributions
LML and MAH conceived the study. MAH and LML conducted the analysis and prepared the draft manuscript. LML, MAH, NCN, and MT reviewed and approved the final version of the manuscript.
Funding
The Canadian Institutes of Health Research (grant # MOP-142404) provided funding for this study. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript. MAH is supported by funding from the Visual and Automated Disease Analytics (VADA) Program at the University of Manitoba. LML is supported by a Tier 1 Canada Research Chair in Methods for Electronic Health Data Quality (CRC-2023-00349).
Data availability
The data was provided to the investigators under specific data-sharing agreements only for approved use at the Manitoba Centre for Health Policy (MCHP), University of Manitoba. The researchers or MCHP does not own the original source data and, as such, cannot be provided to a public repository. The original source data are available through the submission of appropriate ethics and data access approval forms to the Health Research Ethics Board of the University of Manitoba (see https://umanitoba.ca/research/opportunities-support/ethics-compliance for more details) and the Provincial Health Research Privacy Committee (see https://www.rithim.ca/phrpc-overview for more details).
Declarations
Ethics approval and consent to participate
This research was approved by the University of Manitoba Health Research Ethics Board (ethics # HS25905-H2023:077). Participant consent was not required since this study performed a retrospective review of electronic health records, and obtaining consent was impractical, as outlined under Sect. 24 [3] c of the Personal Health Information Act of Manitoba. The Health Research Ethics Board at the University of Manitoba waived informed consent to participate.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Harbaugh CM, Cooper JN. Administrative databases. Semin Pediatr Surg. 2018;27(6):353–60. [DOI] [PubMed] [Google Scholar]
- 2.Barrett JK, Sweeting MJ, Wood AM. Chapter 2 - Dynamic Risk Prediction for Cardiovascular Disease: An Illustration Using the ARIC Study. In: Srinivasa Rao ASR, Pyne S, Rao CR, editors. Handbook of Statistics [Internet]. Elsevier; 2017. pp. 47–65. (Disease Modelling and Public Health, Part A; vol. 36). Available from: https://www.sciencedirect.com/science/article/pii/S0169716117300044.
- 3.Tobias MI, Cavana RY, Bloomfield A. Application of a system dynamics model to inform investment in smoking cessation services in New Zealand. Am J Public Health. 2010;100(7):1274–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective: A research methods review. Soc Sci Med. 2021;272:113697. [DOI] [PubMed] [Google Scholar]
- 5.McGowan VJ, Buckner S, Mead R, McGill E, Ronzi S, Beyer F, et al. Examining the effectiveness of place-based interventions to improve public health and reduce health inequalities: an umbrella review. BMC Public Health. 2021;21(1):1888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Lix L, Puchtinger R, Reimer K, Robitaille C, Smith M, Svenson L et al. The Canadian Chronic Disease Surveillance System: A model for collaborative surveillance. Int J Population Data Sci. 2018;3(3). Available from: https://ijpds.org/article/view/433. [DOI] [PMC free article] [PubMed]
- 7.Desai RJ, Solomon DH, Shadick N, Iannaccone C, Kim SC. Identification of smoking using Medicare data–a validation study of claims-based algorithms. Pharmacoepidemiol Drug Saf. 2016;25(4):472–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Mu Y, Chin AI, Kshirsagar AV, Bang H. Data concordance between ESRD Medical Evidence Report and Medicare claims: Is there any improvement? PeerJ. 2018;6:e5284. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wang Y, Sohn S, Liu S, Shen F, Wang L, Atkinson EJ, et al. A clinical text classification paradigm using weak supervision and deep representation. BMC Med Inf Decis Mak. 2019;19(1):1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Pang X, Forrest CB, Lê-Scherban F, Masino AJ. Prediction of early childhood obesity with machine learning and electronic health record data. Int J Med Informatics. 2021;150:104454. [DOI] [PubMed] [Google Scholar]
- 11.van Vuuren CL, van Mens K, de Beurs D, Lokkerbol J, van der Wal MF, Cuijpers P, et al. Comparing machine learning to a rule-based approach for predicting suicidal behavior among adolescents: Results from a longitudinal population-based survey. J Affect Disord. 2021;295:1415–20. [DOI] [PubMed] [Google Scholar]
- 12.Haque MA, Gedara MLB, Nickel N, Turgeon M, Lix LM. The validity of electronic health data for measuring smoking status: A systematic review and meta-analysis. BMC Med Inf Decis Mak. 2024;24(1):33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.McVeigh KH, Newton-Dame R, Chan PY, Thorpe LE, Schreibstein L, Tatem KS, et al. Can electronic health records be used for population health surveillance? Validating population health metrics against established survey data. J Electron Health Data Methods. 2016;4(1):1267. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Luck J, Larson AE, Tong VT, Yoon J, Oakley LP, Harvey SM. Tobacco use by pregnant Medicaid beneficiaries: Validating a claims-based measure in Oregon. Prev Med Rep. 2020;19:101039. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kim HM, Smith EG, Stano CM, Ganoczy D, Zivin K, Walters H, et al. Validation of key behaviourally based mental health diagnoses in administrative data: suicide attempt, alcohol abuse, illicit drug abuse and tobacco use. BMC Health Serv Res. 2012;12(1):18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Benchimol EI, Smeeth L, Guttmann A, Harron K, Moher D, Petersen I, et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 2015;12(10):e1001885. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Smith M, Roos LL, Burchill C, Turner K, Towns DG, Hong SP et al. Health services data: Managing the data warehouse: 25 years of experience at the Manitoba Centre for Health Policy. In: Levy A, Goring S, Gatsonis C, Sobolev B, van Ginneken E, Busse R, editors. Health Services Evaluation. New York, NY: Springer US; 2019. pp. 19–45. (Health Services Research). Available from: 10.1007/978-1-4939-8715-3_3.
- 18.WHO. Anatomical Therapeutic Chemical (ATC) Classification. 2024. Available from: https://www.who.int/tools/atc-ddd-toolkit/atc-classification.
- 19.Government of Canada SC. Census of Population. Available from: https://www12.statcan.gc.ca/census-recensement/index-eng.cfm.
- 20.Leslie WD, MacWilliam L, Lix L, Caetano P, Finlayson GS. A population-based study of osteoporosis testing and treatment following introduction of a new bone densitometry service. Osteoporos Int. 2005;16(7):773–82. [DOI] [PubMed] [Google Scholar]
- 21.Lewiecki EM. Update on bone density testing. Curr Osteoporos Rep. 2005;3(4):136–42. [DOI] [PubMed] [Google Scholar]
- 22.Goel H, Binkley N, Hans D, Leslie WD. Bone density and trabecular bone score to predict fractures in adults aged 20–39 years: a registry-based study. Osteoporos Int. 2023;34(6):1085–91. [DOI] [PubMed] [Google Scholar]
- 23.MCHP Metadata Repository. Term: Personal Health Identification Number (PHIN). Available from: http://mchp-appserv.cpe.umanitoba.ca/viewDefinition.php?definitionID=103292.
- 24.MCHP Metadata Repository. Term: Income Quintiles / Income Quintile. Available from: http://mchp-appserv.cpe.umanitoba.ca/viewDefinition.php?definitionID=102882.
- 25.Havard A, Jorm LR, Lujic S. Risk adjustment for smoking identified through tobacco use diagnoses in hospital data: A validation study. PLoS ONE. 2014;9(4):e95029. [DOI] [PMC free article] [PubMed]
- 26.Roper L, Tran DT, Einarsdóttir K, Preen DB, Havard A. Algorithm for resolving discrepancies between claims for smoking cessation pharmacotherapies during pregnancy and smoking status in delivery records: The impact on estimates of utilisation. PLoS ONE. 2018;13(8):e0202999. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Marrie RA, Tan Q, Ekuma O, Marriott JJ. Development of an indicator of smoking status for people with multiple sclerosis in administrative data. Multiple Scler J – Experimental Translational Clin. 2022;8(1):20552173221074296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.tidymodels - Tidymodels packages. Available from: https://www.tidymodels.org/packages/.
- 29.Allam A, Nagy M, Thoma G, Krauthammer M. Neural networks versus Logistic regression for 30 days all-cause readmission prediction. Sci Rep. 2019;9:9277. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Uddin S, Imam T, Hossain ME, Gide E, Sianaki OA, Moni MA, et al. Intelligent type 2 diabetes risk prediction from administrative claim data. Inform Health Soc Care. 2022;47(3):243–57. [DOI] [PubMed] [Google Scholar]
- 31.Hastie T, Tibshirani R, Friedman J. Random Forests. In: Hastie T, Tibshirani R, Friedman J, editors. The elements of statistical learning: Data mining, inference, and prediction. New York, NY: Springer; 2009. pp. 587–604. (Springer Series in Statistics). Available from: 10.1007/978-0-387-84858-7_15.
- 32.Tibshirani R. Regression shrinkage and selection via the Lasso. J Royal Stat Soc Ser B (Methodological). 1996;58(1):267–88. [Google Scholar]
- 33.Feurer M, Hutter F. Hyperparameter Optimization. 2018. Available from: https://www.semanticscholar.org/paper/Chapter-1-Hyperparameter-Optimization-Feurer-Hutter/1a90075b632c8cb7762a337431f143c6fe682bad.
- 34.Khan Z, Gul A, Perperoglou A, Miftahuddin M, Mahmoud O, Adler W, et al. Ensemble of optimal trees, random forest and random projection ensemble classification. Adv Data Anal Classif. 2020;14(1):97–116. [Google Scholar]
- 35.Kuhn M. Building predictive models in R using the caret package. J Stat Softw. 2008;28:1–26.27774042 [Google Scholar]
- 36.Fluss R, Faraggi D, Reiser B. Estimation of the youden index and its Associated cutoff Point. Biom J. 2005;47(4):458–72. [DOI] [PubMed] [Google Scholar]
- 37.Monaghan TF, Rahman SN, Agudelo CW, Wein AJ, Lazar JM, Everaert K, et al. Foundational statistical principles in medical research: Sensitivity, Specificity, Positive Predictive Value, and Negative Predictive Value. Med (Kaunas). 2021;57(5):503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 1988;44(3):837–45. [PubMed] [Google Scholar]
- 39.Teramukai S, Okuda Y, Miyazaki S, Kawamori R, Shirayama M, Teramoto T. Dynamic prediction model and risk assessment chart for cardiovascular disease based on on-treatment blood pressure and baseline risk factors. Hypertens Res. 2016;39(2):113–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Chang JT, Meza R, Levy DT, Arenberg D, Jeon J. Prediction of COPD risk accounting for time-varying smoking exposures. PLoS ONE. 2021;16(3):e0248535. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.van Hoeven LR, de Bruijne MC, Kemper PF, Koopman MMW, Rondeel JMM, Leyte A, et al. Validation of multisource electronic health record data: An application to blood transfusion data. BMC Med Inf Decis Mak. 2017;17(1):107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Reps JM, Rijnbeek PR, Ryan PB. Supplementing claims data analysis using self-reported data to develop a probabilistic phenotype model for current smoking status. J Biomed Inform. 2019;97:103264. [DOI] [PubMed] [Google Scholar]
- 43.Ali S, Na R, Waterhouse M, Jordan SJ, Olsen CM, Whiteman DC, et al. Predicting obesity and smoking using medication data: A machine-learning approach. Pharmacoepidemiol Drug Saf. 2022;31(1):91–9. [DOI] [PubMed] [Google Scholar]
- 44.Choi J, Jung HT, Ferrell A, Woo S, Haddad L. Machine learning-based nicotine addiction prediction models for youth e-cigarette and waterpipe (Hookah) users. J Clin Med. 2021;10(5):972. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Kroeker K, Widdifield J, Muthukumarana S, Jiang D, Lix LM. Model-based methods for case definitions from administrative health data: Application to rheumatoid arthritis. BMJ Open. 2017;7(6):e016173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Tate C, Kumar R, Murray JM, Sanchez-Franco S, Montgomery SC, Montes F, et al. Socio-environmental and psychosocial predictors of smoking susceptibility among adolescents with contrasting socio-cultural characteristics: A comparative analysis. BMC Public Health. 2021;21(1):2240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Leslie WD, Morin SN, Martineau P, Bryanton M, Lix LM. Factors associated with bone density monitoring while on antiosteoporosis treatment in routine clinical practice: A registry-based cohort study. J Clin Densitometry. 2020;23(4):568–75. [DOI] [PubMed] [Google Scholar]
- 48.Agarwal A, Baleanu F, Moreau M, Charles A, Iconaru L, Surquin M, et al. External validation of FRISBEE 5-year fracture prediction models: A registry-based cohort study. Archives Osteoporos. 2022;18(1):13. [DOI] [PubMed] [Google Scholar]
- 49.Galar M, Fernandez A, Barrenechea E, Bustince H, Herrera F. A review on ensembles for the class imbalance problem: Bagging, Boosting, and Hybrid-Based approaches. IEEE Trans Syst Man Cybernetics Part C (Applications Reviews). 2012;42(4):463–84. [Google Scholar]
- 50.Pezoulas VC, Zaridis DI, Mylona E, Androutsos C, Apostolidis K, Tachos NS, et al. Synthetic data generation methods in healthcare: A review on open-source tools and methods. Comput Struct Biotechnol J. 2024;23:2892–910. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Juwara L, El-Hussuna A, El Emam K. An evaluation of synthetic data augmentation for mitigating covariate bias in health data. Patterns. 2024;5(4):100946. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Feely A, Lim LS, Jiang D, Lix LM. A population-based study to develop juvenile arthritis case definitions for administrative health data using model‐based dynamic classification. BMC Med Res Methodol. 2021;21:105. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data was provided to the investigators under specific data-sharing agreements only for approved use at the Manitoba Centre for Health Policy (MCHP), University of Manitoba. The researchers or MCHP does not own the original source data and, as such, cannot be provided to a public repository. The original source data are available through the submission of appropriate ethics and data access approval forms to the Health Research Ethics Board of the University of Manitoba (see https://umanitoba.ca/research/opportunities-support/ethics-compliance for more details) and the Provincial Health Research Privacy Committee (see https://www.rithim.ca/phrpc-overview for more details).