An Introduction to Machine Learning for Clinicians: How Can Machine Learning Augment Knowledge in Geriatric Oncology?

Introduction

Interest in machine learning (ML) approaches to analyze patient data is in an explosive phase of growth. Underpinning this burgeoning interest are several advances in technology, including availability of computing power, evolution of ML software, and the ubiquity of electronic health records (EHRs). Personal computers now have sufficient computational power to run some ML algorithms for small to medium datasets, encompassing many of the datasets of interest in clinical medicine. Even for very large datasets (“big data”, with number of data points in the billions, trillions, or more), access to parallel computing resources is now widespread at academic medical centers and via cloud computing platforms. Advances in ML software maturity have substantially lowered the expertise threshold for ML. Many applications now exist, several of them free and open-source, assisting people at all skill levels to analyze their data using ML algorithms; code libraries to run ML algorithms are also available for use in Python,¹ R,² SAS,³ Stata,⁴ MATLAB⁵, and other programming languages. Importantly, most healthcare data are now stored digitally, within EHRs and other digital databases; in theory, these data may therefore be more easily collated and fed to ML algorithms on a large scale. The increasing adoption of digital technologies to collect patient data, such as wearables and online questionnaires, will only increase the appetite for ML implementation.⁶ These advances have led to a growing number of ML applications across a number of fields, yielding insights that would have previously been unobtainable or very difficult to obtain using “traditional” statistical methods.^7,8 Analyses of healthcare datasets using ML are emerging with increasing frequency in the literature; within the realm of cancer research, ML approaches are driving deeper understanding of cancer risk, diagnosis, prognosis, and treatment (Table 1).

Table 1.

Example Studies Using Machine Learning in Cancer Research.

Topic	Example Study	How did ML advance understanding?	Additional examples
Cancer risk	Comparison of machine learning risk model versus a validated clinical risk tool (BOADICEA) to calculate breast cancer risk using data from 112,587 patients27	Supervised learning models (Markov Chain Monte Carlo generalized linear mixed models, Random Forest, AdaBoost) better predicted breast cancer risk compared to clinical decision tool (AUC 0.84 – 0.89 vs 0.64). ML models reclassified >35% of women into different breast cancer risk categories, mostly into higher-risk categories entailing more aggressive screening.	Yala et al 2019,28 Nartowt et al 2019,29 Perera et al 202030
Diagnosis	Development of a DNA methylation-based classification model for central nervous system tumor diagnosis, using 2,801 reference samples31	Clustering (unsupervised learning) was used to understand DNA methylation profiles of tumors and relationship to histology. Application of a Random Forest (supervised learning) model changed the diagnosis in 12% of cases, compared to histologic diagnosis, with crucial implications for prognosis and treatment.	Wang et al 2019,23 Yamada et al 2019,32 Tschandl et al 201933
Prognosis	Development of prognostic classification models for 5- year overall survival after surgery for pancreas cancer, using tissue samples from 194 patients34	Hierarchical clustering (unsupervised learning) was used for initial understanding of gene expression data. Support vector machine (SVM) and neural network models (both supervised learning methods) were better able to predict prognosis based on tissue gene expression than regression models.	Chen et al 2007,35 Wang et al 2011,36 Jiang et al 2018,37 Huang et al 201938
Treatment	Development of predictive models for response to chemotherapy (gemcitabine [gem] and 5-fluorouracil [5- FU]) based on gene expression data from patients’ primary tumor tissue (n=92 patients, expression levels for 60,483 genes)39	Multiple clustering (unsupervised learning) algorithms compared for the gene expression variables, to reduce the number of independent variables (dimension reduction) to 32 clusters for 5-FU and 50 for gem. Prediction of chemotherapy response was as high as 86% in validation cohorts; Random Forest and SVM models out-performed logistic regression.	Mucaki et al 2019,40 Cain et al 2018,41 Huang et al 201742

ML is generally used for classification or clustering of items that group together based on features (or variables) in large datasets. ML models can be utilized to identify patterns in data, or to build predictive models to be used for real-time outcome prediction or decision support. Advantages of ML include the ability to create models that mix multiple variable classes (i.e., continuous, categorical, and binary variables can be used within the same model), enhanced techniques for imputation of missing data, and in many cases, higher predictive precision compared to traditional predictive models. However, there is a well-described tradeoff in ML between predictive power (flexibility), and explanatory power (interpretability), both of which are crucial considerations for clinical questions. ML models with the highest predictive power are generally opaque with respect to identifying causal associations and explanatory mechanisms. High predictive power is desirable in the oncology clinic for accurate forecasting of clinical outcomes (e.g. survival, treatment benefit, tolerability, quality of life) and reducing patient uncertainty; however, interpretability may be necessary for pinpointing factors amenable to intervention and enhancing clinicians‟ acceptance and uptake of the model.

ML methods may be particularly suitable for analyzing data from older adults with cancer. By 2030, 70% of all cancer diagnoses will occur in people >65 years of age,⁹ but this same group comprises only 40% of participants in clinical trials which establish the standards for cancer treatment.¹⁰ Typically, older participants in clinical trials are the most “fit” representatives of their age groups, and therefore using the randomized clinical trial paradigm to guide therapy decisions will not sufficiently represent older adults with competing risks. Older adults have many potentially competing risks for adverse outcomes, creating difficulties in analyzing how treatments affect outcomes like overall survival within usual clinical trial models.¹¹ Moreover, identifying the numerous variables impacting outcomes in older adults with cancer (including patient-reported outcomes, or PROs) has been the focus of extensive prior work in geriatric oncology,¹² but these variables are often not captured or analyzed for clinical trials.

Pragmatic datasets such as EHR-abstracted data merged with patient-reported outcomes (PROs), and analyzed using novel ML methods, may overcome these knowledge limitations, including older, frailer adults not suitable for clinical trials, and broadening the set of available variables. As mentioned above, ML methods excel at mixing of categorical variables often found in PROs with continuous variables found in EHRs (e.g. weight, white blood cell count, hemoglobin) where traditional statistical methods may struggle. Importantly, in assembling these datasets, particularly those with high dimensionality (a large number of variables), data cleaning, model analysis, and model interpretation are critical and must be accomplished with care and expertise so as not to overstate conclusions. Successful blending of ML techniques with large pragmatic datasets may complement, and perhaps even overtake, clinical trials as a source of knowledge in this population. Additionally, these methods could permit better characterization and understanding of patient trajectories,¹³ leading to better tools to help with shared decision-making between oncologists and patients.

This primer introduces a series of methods articles about the use of ML approaches in geriatric oncology datasets. The purpose of these articles is to explain the steps in application of ML, discuss differences with traditional statistical approaches, and to compare model outputs in terms of flexibility, accuracy, and interpretability. Two subsequent papers will examine the use of ML to predict falls in older adults on chemotherapy, and to identify groupings of PROs that may co-occur and/or have predictive importance in this population.

Machine Learning: A brief primer

Machine learning algorithms are generally categorized as either supervised learning (usually known as „classification‟) or unsupervised learning (of which a common application is “clustering”). Supervised learning algorithms, such as k-nearest neighbor, random forest, and artificial neural networks, are trained on datasets with known, labeled outcomes of interest (e.g. frailty, survival). Supervised ML algorithms infer decision rules about how to assign or predict the class labels of new data points; for this reason, regression is often considered to be a supervised learning method, as it yields a model which can be used to classify or predict output from new sets of data points. Availability of validated (i.e. ground truth) labelled data is a requirement for supervised learning methods to succeed. For example, we trained supervised learning models to classify older adults who fell within 3 months of starting chemotherapy versus non-fallers (i.e., “faller” versus “non-faller” was the class label), using geriatric assessment, demographic, cancer-related, and other variables as model input. Description of the approach and model output will be the focus of the next paper in this methods series.

Unsupervised learning models explore unlabeled data to uncover the underlying data structure. It is most often used to detect previously unknown patterns within datasets. Most unsupervised ML algorithms are related to „clustering‟, which attempts to define clusters of data points which are “closer” or more similar to each other. This is easily accomplished and visualized in two or three dimensions (Figure 1), but given a high number of variables (and therefore dimensions), more sophisticated and computationally intensive algorithms¹⁴ and visualization methods¹⁵ are required. Unsupervised methods are often used in an exploratory way, rather than to yield definitive conclusions, and output is highly dependent on the algorithm and hyperparameters selected. Hyperparameters, also called tuning parameters, are values used to control the behavior of the ML algorithm (e.g., number of clusters, distance or density thresholds, type of linkage between clusters). Algorithms exist to detect clusters based on spatial distance between data points, space or subspace density, network connectivity between data points, or other measures; each of these techniques may “detect” markedly different cluster groupings. Biological taxonomies, such as species taxonomies, were developed using early conceptual precursors to unsupervised ML, and taxonomy problems today are often tackled using unsupervised ML. Within the medical field, clustering has been used in drug discovery,¹⁶ genomic analysis,¹⁷ and analysis of medical images,¹⁸ among other applications. The application of unsupervised ML to patient-reported data is in its infancy, but we have applied unsupervised ML algorithms to patient-reported symptom data to try to detect patterns within a dataset of older adults with advanced stage cancer receiving chemotherapy.¹⁹ This example will be the focus of a subsequent paper in this methods series.

Figure 1.

Visualizing clusters in 2 dimensions.

Supervised and unsupervised ML can be used effectively in tandem or in sequence depending on the particular dataset and research question. For example, unsupervised ML could be used to detect latent classes of patients, and then supervised learning models could be trained using these latent classes as the class labels. “Semi-supervised” approaches use a combination of the two techniques; for example, if labelling data points is resource-intensive, a few could be labelled while most remain unlabeled, and a semi-supervised approach would be valid. Other techniques under the rubric of ML include time series forecasting, with particular utilization by businesses and industries that analyze financial and sales data. Most healthcare data, including data obtained from patients with cancer, have a temporal component, and an understanding of trajectory is often a crucial component of cancer treatment decision-making. This may be particularly true for older adults, who often value knowledge about their anticipated functional, cognitive, or symptomatic trajectories above knowledge about survival outcomes.²⁰

Machine Learning for Geriatric Oncology: Drawbacks and Challenges

Despite the promise entailed by ML methods for analyzing highly multivariable and/or longitudinal data collected from older adults with cancer, several limitations should temper enthusiasm and prompt careful scrutiny of these methods. For the foreseeable future, the lack of EHR interoperability and collaborative, shared data architecture may limit the assembly of datasets sufficiently large for ML analysis. Similar to traditional statistical methods like linear regression (a ML model itself), a sufficient sample size is necessary for input to ML algorithms, to minimize the competing risks of bias (causing underfitting of the data) and variance (causing overfitting). ML models can be “data hungry”: although heuristics to estimate necessary sample size are controversial, it is estimated that some common ML algorithms may require more than 10 times the number of samples per variable compared to linear regression models.²¹ Even the largest datasets typically collected in prospective clinical trials, with sample sizes in the low hundreds, may be insufficient for ML analysis; other large databases like SEER-Medicare do not collect all data relevant to older adults (such as PROs and geriatric assessment data). Large prospective datasets in geriatric oncology are rare. In order to address this problem, multicenter prospective data collection of the relevant variables is needed. Attempts to define a standardized “minimum data set” for older adults with cancer, such as recent American Society of Clinical Oncology (ASCO) guidelines, can facilitate the development of large datasets.²² These data should optimally be maintained in a centralized, collaboratively resourced data warehouse with standardized intake and validation procedures.

Even with large datasets, and despite the availability of user-friendly software packages allowing anyone to apply ML methods, significant expertise is required to prepare the data for analysis, select and tune the appropriate ML algorithm, implement the model efficiently within the available computational infrastructure, and assess for potential bias. Data validation, data transformation (such as selecting variable type and normalizing variable values), and feature selection (choosing the variables to include in the model) require considerable time and experience. A variety of ML algorithms are available, and can yield significantly different output with the same training data set. Even within algorithms, model tuning (i.e., selecting hyperparameters) is not standardized and is based largely on expertise and experience. Unlike with regression methods, wherein the statistical model can be examined variable by variable, many ML algorithms are “black boxes,” with inscrutable behavior generating the model output. Even when the inner workings of the model can be inspected, as in regression, or with recently developed mechanisms to examine the inner workings of neural network models,^23,24 it can be very difficult even for experts in these methods to identify, describe, and correct for sources of bias. Ethically, it is crucial that research teams have the skill and dedication to rigorously examine, validate, and question the reliability and completeness of data input into ML models, particularly when the workings of the model cannot be easily parsed. Failure to eliminate sources of bias in the model training data can result in meaningless (at best) or highly destructive (at worst) conclusions.²⁵

The role of “data scientist” is a relatively new concept, combining the totality of skills needed to work with data across its life-cycle (Figure 2). Among these skills are knowledge of data architecture (such as databases), programming ability, statistical knowledge (including deep understanding of ML algorithms), proficiency with data visualization, and content expertise. Unfortunately, although approximately 30% of stored data in the world is healthcare data, only 3% of data scientists in the United States work within the healthcare sector.²⁶ It is estimated, by benchmarking with other data-driven industries, that 10 to 20 times more data scientists are needed in healthcare.²⁶ The demand for these professionals far outpaces their availability, but their skills could greatly augment the analysis of complex data in older adults with cancer. Research teams in geriatric oncology should consider the addition of data scientists to their teams; at the University of Rochester, rich collaborations have formed between the Geriatric Oncology Research Group and the Goergen Institute for Data Science, yielding work on ML methods applied to geriatric oncology datasets.

Figure 2.

Data Science lifecycle.

Conclusion

Use of ML algorithms in the analysis of healthcare data is increasingly common, and clinicians should be familiar with basic ML concepts. Application of ML to understand trajectory and outcomes in older adults with cancer is intriguing, albeit with several constraints including small currently available clinical datasets and limited workforce with ML expertise within healthcare. These constraints have limited adoption of both “traditional” ML algorithms (the focus of this paper) as well as more sophisticated methods such as deep learning which are being pioneered in other industries. Deep learning methods can enhance model performance, but require massive datasets and computational power which are out of reach for most clinical applications at the current time (Figure 3).

Figure 3.

Model performance as a function of amount of data for traditional ML and deep learning algorithms.

This introduction to basic ML concepts and methods is the first in a series of articles illustrating ML algorithms applied to clinical datasets of older adults with cancer. The goal of this series is to encourage the discussion of these methods as possible adjuncts to traditional regression models, and in turn to increase understanding of the data most crucial for decision-making for older adults with cancer and their care teams.