An Introduction to Machine Learning for the Pediatric Hospitalist

Abstract

Machine learning models are increasingly used in clinical research to predict patient outcomes, yet many clinicians lack the training to critically appraise these studies. This article provides a conceptual introduction to machine learning for the pediatric hospitalist with no prior computational experience. We focus on the most common application in clinical medicine: supervised learning, where models learn from data with known outcomes to make predictions about new unseen patients. Core tasks such as classification and regression are explained, along with intuitive models like decision trees and advanced methods like ensembles. Essential concepts for critical appraisal, including overfitting and leakage, the challenge of interpretability, and data bias, are highlighted. We emphasize the importance of model validation and the distinction among prediction, interpretation, and causation. The article concludes by deconstructing a published pediatric study to illustrate these principles in practice, equipping the reader to better understand and evaluate research that uses machine learning. Our goal is to equip pediatric hospitalists with the foundational knowledge to become informed consumers and potential contributors within the machine learning ecosystem, ensuring that this technology augments, rather than replaces, clinical judgment.

INTRODUCTION

The landscape of inpatient medicine is being transformed by the unprecedented expansion of clinical data. The widespread adoption of electronic health records, advances in medical imaging, continuous physiologic monitoring, and the integration of genomic information have created vast, complex datasets that far exceed the capacity of humans to process them without computational help.¹ This data-rich environment presents both a challenge and an opportunity: although the volume of information can overwhelm traditional clinical reasoning, it also opens the door to new analytical approaches that are capable of revealing previously inaccessible patterns in clinical data.²

Machine learning (ML), a branch of artificial intelligence (AI), is a powerful tool for making sense of this complexity. By learning directly from data, ML models can identify subtle relationships to support diverse clinical tasks such as risk prediction and diagnostic assistance.³ However, the rapid proliferation of ML-based research presents a new challenge for most clinicians: how to critically evaluate these studies. This article aims to give pediatric hospitalists a framework for understanding the core concepts of ML. By equipping hospitalists with the tools to appraise ML-based research, we hope to foster an informed, evidence-based approach to these emerging technologies.⁴

WHAT IS ML? THE CORE IDEA OF LEARNING FROM DATA

Supervised Learning: Learning With an Answer Key

Supervised learning is the most common and intuitive form of ML, serving as the workhorse for clinical prediction tasks.⁵ The core idea is analogous to training a person. Imagine showing the student a series of chest x-rays, each labeled as either “pneumonia” or “no pneumonia.” Over time, the student learns to associate specific visual patterns with the correct diagnosis. In supervised learning, the computer algorithm is the student, and the dataset of labeled examples is the study material.

In the language of ML, each example consists of features (also called covariates, independent variables, or predictors in statistical language) and a label (also called target or dependent variable in statistical language). Features are the measurable inputs, such as a patient’s vital signs, laboratory results, or demographic information. The label is the known outcome we want the model to predict, for instance, whether the patient developed sepsis. The algorithm learns by analyzing many of these feature-label pairs to identify the relationships between them.

Supervised learning tasks in clinical research generally fall into 2 categories: classification and regression.

• Classification is used to predict a binary (ie, there are 2 options) or multinomial (ie, there are >2 options) categories. For example, a classification model might predict transfer to the intensive care unit (ICU) or not.

• Regression is used to predict a quantitative numerical value. For example, a regression model might predict the length of stay of a patient.

Unsupervised Learning: Finding Patterns Without a Label

Unsupervised learning analyzes data without labels or known outcomes. For example, instead of being told which chest radiographs show pneumonia, the model receives many radiographs with no associated diagnoses and looks for natural groupings or patterns. Although this approach is useful for data exploration and hypothesis generation and includes the methods used in the recent generative AI revolution, the remainder of this article focuses on supervised learning, which dominates clinical prediction work.

A GLIMPSE AT COMMON MODELS: FROM LINES TO FORESTS

The field of supervised ML offers a diverse toolkit of models. To start, there are linear models, a category that includes the simple linear regression model (y = mx + b) as well as its generalizations like the logistic regression model commonly seen in clinical research. These models, part of a family known as generalized linear models,⁶ are important to researchers primarily for their simplicity and clear interpretability; an interpretable model is one in which the relationship between the feature and the label can be easily and clearly understood. For example, in a linear regression model, the effect of each feature is represented by a single coefficient, which quantifies the expected change in the label for a 1-unit change in that feature, holding all other features constant. This transparency allows researchers to draw meaningful inferences about which variables are associated with the label and is especially valuable when the goal is to understand the underlying mechanisms driving a clinical phenomenon.

However, clinical data often contain complex, nonlinear relationships that linear models cannot capture. ML expands modeling to fit nonlinear patterns through a variety of flexible model types. Perhaps the simplest example is the decision tree. A single decision tree is a model that closely mirrors clinical reasoning. It operates like a flowchart, asking a series of yes or no questions about a patient’s information to arrive at a prediction.⁷ For example, a decision tree might first ask, “Is the patient’s temperature >38 °C?” followed by, “Is the white blood cell count >15 000?” Each answer guides the model down a specific path, ending in a final prediction. The primary advantage of a single decision tree is its high interpretability; any prediction can be traced back through the simple, explicit rules it followed.

To fit the wide array of nonlinear data, there are many additional model types (Table 1). There are also ensemble methods that group multiple individual models together, with decision tree ensembles being the most common. Broadly speaking, these methods progress from highly interpretable to minimally interpretable as they become more flexible to nonlinear patterns and each method has important nuances for their application.

TABLE 1.

Common Supervised Machine Learning Methods

Model Family	What It Does	When It Shines	Interpretability	Notes for Clinical Use
Generalized linear models (eg, logistic regression)	Fits a linear relationship between predictors and outcome	Small to medium structured clinical data that can be arranged in tables (eg, vital signs or laboratory test results) rather than images or text; need clear effect estimates	High	Coefficients map to odds or risk; watch for multicollinearity, particularly if linear imputation was used to fill missing data
Decision tree	Splits data with if/then rules on single variables	When data can be arranged in tables; needing easy bedside explanation	High	Unstable to small data changes
k-Nearest neighbors	Predicts from the most similar patients in the data	Finding local patterns with variables that have similar scales	Low-medium	Sensitive to scaling and class imbalance; slow at large scale
Support vector machines	Finds a boundary that maximizes separation between groups	Classifying complex groups using many variables, but limited samples	Low	Requires feature scaling and kernel tuning; hard to explain at bedside
Naive Bayes	Combines simple per-feature likelihoods assuming independence	Text, when a simple baseline model is helpful, quick prototypes	Medium	Fast and robust; independence assumption often violated but can still work
Neural networks	Stacks layers to learn complex functions	Data with many variables; large samples	Low	Powerful but harder to explain; needs careful regularization and monitoring
Tree ensembles: random forest	Averages many decision trees grown on resampled data	Good default for EHR data that can be arranged in tables (ie, not images or text)	Medium	Robust, handles nonlinearities and interactions; variable importance is relative
Tree ensembles: gradient boosting	Adds small decision trees sequentially to correct prior errors	Often top performance on data that can be arranged in tables (ie, not images or text)	Medium	Needs tuning and careful validation; monitor calibration

Abbreviation: EHR, electronic health record.

In many cases, simple nonlinear models like decision trees can be unstable; small changes in the training data can result in large changes in the model and its predictions. To improve performance, ML often relies on ensemble models, which follow the principle of “the wisdom of the crowd.” For example, instead of building 1 tree, an ensemble method might combine the predictions from hundreds or even thousands of decision trees. In addition, despite the advancements in transformer-based generative AI models, decision tree–based ensemble methods, in general, are still often the best performing models for typical tabular datasets (ie, data that can be represented by a spreadsheet as opposed to images or sounds) and most time series forecasting applications.⁸ Two popular ensemble techniques are random forests (which build many trees on varying subsets of the data and average their predictions)⁹ and gradient-boosted machines (which build trees sequentially, with each new tree learning from the errors of the previous one).¹⁰

Modern applications of gradient-boosted machines typically use specific implementations such as XGBoost, LightGBM, or CatBoost that each have subtle distinctions in their approach to tree-building, handling of categorical variables, and overfitting prevention. Although complex ensemble models historically faced resistance because of concerns about interpretability, recent advancements have significantly improved these challenges for decision tree–based ensemble methods.¹¹

A CLINICIAN’S GUIDE TO CRITICAL APPRAISAL

When reading a study that uses ML, the pediatric hospitalist should not focus on the mathematical details of the algorithm but rather on the study’s design, data, and validation. It is important to note that ML models are fundamentally designed to identify complex associations in data to make predictions. They are not designed to establish causality. This is a critical distinction. They cannot, on their own, be used to decide on a course of treatment, as acting on a correlation as if it were a causal relationship can lead to patient harm.¹² Beyond the causal caveat, there are a few fundamental questions to guide the critical appraisal of any clinical prediction model. For a complete checklist of fundamental issues when evaluating a manuscript using ML, the TRIPOD+AI list is available.^13,14

The Most Important Question: Was the Model Validated on Unseen Data?

The single most important concept in ML is validation. To provide flexibility, these models do not make many assumptions about the statistical distribution of the data. As a result, there is no sense of SEs, CIs, or statistical testing directly from the model. Thus, a model’s performance can only be meaningfully evaluated on data it has never encountered. To achieve this, researchers typically split their dataset into at least 2 parts: a training set and a test set. The model learns patterns exclusively from the training data. The test data are kept completely separate and unseen until the model is finalized. This test set serves as the final examination, providing an honest assessment of how the model is likely to perform in the real world on new patients.^13,14

This separation is critical to avoid 2 related problems: overfitting and data leakage. Overfitting occurs when a model memorizes the training data, including noise and idiosyncrasies, rather than learning generalizable patterns. Data leakage occurs when information that would not be available at prediction time enters the training pipeline (eg, including laboratory tests drawn after the time when a prediction would be made or allowing duplicate encounters for the same patient with 1 being in the training set and the other in the test set). A model that is overfit or affected by leakage may appear to perform well in development but will fail to generalize. When one sees performance metrics such as the area under the receiver operating characteristic curve (AUROC), the first question to ask is whether the results came from a separate, previously unseen test set without leakage. If not, the reported performance is likely an overestimate of true clinical value.

The Quality Check: Garbage In, Garbage Out

The principle of “garbage in, garbage out” is paramount in ML. A model reflects the data it was trained on, and it will faithfully learn any flaws, gaps, or biases present in that data. Moreover, current clinical information systems were primarily designed for priorities other than clinical or biomedical research (primarily for billing), and therefore they often do not serve as a reliable data source for ML applications. If a dataset contains systematic errors in measurement or documentation, the model will incorporate those errors into its logic. More insidiously, models can learn and even amplify existing societal and clinical biases. For example, if a particular demographic group has historically had less access to care, leading to different data patterns, a model could mistakenly learn that this group is inherently at lower or higher risk, perpetuating health inequities.^15,16

For the practicing hospitalist, this principle has a very practical implication: data representativeness. The source and context of the data are everything. Before trusting a model’s predictions, one must be confident that it was trained on data that are a fair and accurate representation of the population of a specific patient.

Additional Issues

It is important to beware of 4 additional issues when judging results. First, when outcomes are rare, AUROC can look strong even when the positive predictive value is low; investigators should report positive and negative predictive value at clinically meaningful thresholds and include the area under the precision-recall curve to partially address this issue.¹⁷ Second, calibration is critical. Calibration means that predicted risks match observed rates; for example, among patients assigned a 10% predicted risk, approximately 10% should experience the outcome in real data. It is important to assess calibration because poorly calibrated models undermine bedside decisions even when they perform well for point estimates.^18,19 Third, it is important to avoid metric cherry picking by prespecifying the primary metric and the operating threshold that match the intended clinical decision.¹⁴ Fourth, models can drift not only over time but also across sites and populations; external validation should include both temporal and geographic cohorts to assess generalizability.¹⁹

REAL-WORLD APPLICATION: AN EXAMPLE IN HOSPITAL PEDIATRICS

To see these principles in practice, consider a recent study by Foote et al in Hospital Pediatrics that developed an ML model to predict clinical deterioration in hospitalized children.²⁰

The study’s goal was a classic supervised learning task: classification. The authors aimed to predict a binary outcome: whether a patient would experience clinical deterioration, defined as an unplanned ICU transfer within 24 hours or inpatient mortality within 48 hours. The researchers developed and compared 2 ensemble models, a LightGBM and a random forest, against the hospital’s existing institutional Pediatric Early Warning Score (I-PEWS). The models used hundreds of features from the electronic health record, including vital signs, laboratory results, and comorbidities.

The study employed a robust validation method to avoid overfitting. The authors first trained their model on a large dataset of patient encounters but held back 10% of these data as a separate internal test set. More importantly, they performed a temporal validation, testing the final model on a completely new dataset of patients admitted in a subsequent year. This approach provides a strong, real-world assessment of how the model would perform on new patients.

The results illustrate a key trade-off in clinical prediction. The LightGBM model was significantly better at discriminating between patients who would and would not deteriorate (an AUROC of 0.785 vs 0.708 for I-PEWS in the temporal cohort) and provided a much earlier warning (a median lead time of 11 hours vs 3 hours). However, this increased sensitivity came at the cost of a lower positive predictive value. At a clinically relevant threshold, the model would generate approximately 17 false-positive alerts for every 1 true deterioration event, compared with approximately 3 per true event for I-PEWS, raising concerns about alarm fatigue.

Finally, the authors appropriately acknowledged the model’s limitations, which directly reflect our critical appraisal principles. As a single-center study, its generalizability to other hospitals with different patient populations is unknown, highlighting the importance of data representativeness. This example demonstrates how a well-designed ML study is transparent not only about its strengths but also about the practical barriers to clinical implementation.

CONCLUSION

ML holds significant promise for pediatric hospital medicine, offering powerful new ways to analyze complex clinical data for tasks like risk prediction. However, these models are not infallible. Their value is entirely dependent on rigorous development and thoughtful clinical evaluation. As this article has outlined, a model’s utility hinges on proper validation on unseen data, a clear understanding of the data’s limitations and biases, and the critical distinction between prediction and causation. We should be skeptical when studies report only training performance without validation on a test set, provide a vague description of data collection and cleaning, or suggest clinical decision-making without acknowledging that correlation does not equal causation.

Ultimately, ML should be viewed as a powerful tool, not a substitute for clinical expertise. The goal is not for every hospitalist to become a data scientist but to improve ML literacy and allow pediatric hospitalists to join this rapidly expanding field (Table 2). By asking the right questions, clinicians can ensure that these advanced analytical methods are used to augment their own clinical judgment for the benefit of their patients.

TABLE 2.

Take-Home Points

Core Principle	Take-Home Point for the Clinician
1. Defining supervised learning	This is the workhorse of clinical prediction. The model learns from data with known outcomes (eg, ICU transfer: yes/no) to make predictions about new, unseen patients
2. Validation is paramount	A model’s true performance can only be judged on data it has never seen before. Always ask: “Was the model validated on a separate test set?” If not, the reported accuracy is likely an overestimation and not reflective of real-world performance
3. Garbage in, garbage out	A model is only as good as its training data. It will learn any flaws or biases present in that data. Always consider: “Is the training data representative of my patient population?” A model trained at 1 center may not be generalizable to another
4. Prediction is not causation	Machine learning models are designed to find complex associations, not to prove that one factor causes another. Never use a model’s prediction alone to justify clinical intervention. Acting on a correlation as if it is causal can lead to patient harm
5. High accuracy ≠ clinical utility	A model can be statistically excellent but practically challenging. A high-sensitivity alarm may have a low positive predictive value, leading to alarm fatigue. Consider the practical implications of implementing the model in a real-world clinical workflow
6. Augmentation, not replacement	Machine learning is a powerful tool to support, not replace, clinical expertise. The goal is to augment clinical judgment with new insights from complex data, improving patient care

Abbreviation: ICU, intensive care unit.

ABBREVIATIONS

AI

artificial intelligence

AUROC

area under the receiver operating characteristic curve

ICU

intensive care unit

I-PEWS

institutional Pediatric Early Warning Score

ML

machine learning