Ethical Considerations Related to Using Machine Learning-Based Prediction of Mortality in the Pediatric Intensive Care Unit

Machine learning shows promise for developing prediction models that could improve care in the pediatric intensive care unit (PICU). Advocates claim these systems enhance prognostic accuracy and can adapt to changing clinical practices by adding more and new large-scale child health data. Accurate predictive models using machine learning could benefit decision-making and care delivery and, in turn, outcomes for patients and families. Despite their potential, some of these models may replicate the biases of their training datasets or may be biased in other ways (eg, label bias or contextual bias), and are built without the capacity to explain how they reach decisions (so-called black boxes). Moreover, implicit trust or mistrust in technology may influence patients’, families’, and clinicians’ views of software-generated opinions as more objective and valid than they really are. This essay provides an overview of the ethical concerns posed by the advent of machine learning-based models for mortality prediction in the PICU. We discuss the benefits and risks related to this emerging technology, including considerations of technical questions, care delivery, family experience and decision-making, and clinician-family relationships, as well as legal and organizational issues.

Artificial intelligence (AI) enables computers to perform tasks or reasoning typically associated with human intelligence.¹ Machine learning, a subset of AI, identifies patterns in data which advance understanding of medical problems or diseases, and can make predictions about future health or healthcare needs.² In adults, machine learning can diagnose breast cancer from mammograms and predict cardiovascular risk using retinal fundus photographs.^3,4 Infectious disease experts and epidemiologists use machine learning to risk stratify people for specific infections, understand pathogen-host interactions, and predict the emergence or spread of infectious diseases.² In the adult intensive care unit, machine learning has the potential to identify patients at high risk of circulatory failure.⁵

In the PICU, a use case for machine learning is predicting mortality.⁶ Current PICU mortality prediction tools, such as the Pediatric Risk of Mortality III, Pediatric Index of Mortality 2, and Pediatric Index of Mortality 3, are limited because they rely on admission data or data obtained in the first 24 hours of admission, are less accurate for prolonged PICU admissions among certain subgroups, and are validated for group risk determination rather than for individual patients.^7,8 More accurate PICU mortality prediction tools using machine learning could help to decide management plans, prioritize resource use, or measure quality of care delivery, and in research, adjust for case mix and severity of illness.^7,8 Some specific examples include guiding decision-making about using third- and fourth-line therapies for children whose cancer has not responded to previous treatments; helping clinicians to counsel families about goals of care decisions for patient with chronic medical conditions who are demonstrating progressive clinical deterioration; and better identifying patients potentially eligible for donation after circulatory death.⁹

Despite their potential, ethical concerns related to bias, trust, and the impact on care arise with using machine learning to predict mortality in the PICU. We discuss the ethical challenges associated with applying this technology in PICU patients by considering the benefits and risks related to the technology and to care delivery, as well as organizational and legal issues.

Technical Considerations

Machine learning can efficiently incorporate large datasets, enhancing the accuracy of PICU mortality predictions. Such prediction analysis can be repeated on an ongoing basis during a patient’s admission, allowing for real-time assessments of risk and providing ongoing information for decision support. Further, as data accumulate, machine learning can self-adjust, becoming even more accurate.

Some technical challenges, however, threaten the ethical use of machine learning. First, machine learning relies on “learning” from comprehensive datasets. However, models that rely on data about ICU deaths do not account for death after the hospitalization, creating a kind of reverse survivor-ship bias. Second, a lack of diversity or inaccuracy in datasets become reflected in predictions. Third, an algorithm itself can be biased. For example, a risk prediction tool developed to identify patients for high-risk care management programs used healthcare costs as a proxy for assessing health risk. Using this measure, Black patients assigned the same level of risk as White patients were actually sicker. Bias arose because the data reflected the limited access of Black patients to healthcare (and therefore lower costs) compared with White patients.¹⁰ Fourth, some prediction algorithms are so complex that one cannot determine how decisions are made (the black box phenomenon).¹¹ This lack of transparency can lead to or contribute to mistrust and may affect clinician and patient acceptance and the use of such technology if the models are not checked properly for their safety and effectiveness. Some AI could decrease provider biases by mitigating providers’ systematic (intended or unintended) differences in care delivery based on race, ethnicity, demographics, or other characteristics. Balancing the benefits with risks related to bias is important. Finally, if providers become dependent on AI, contingency plans are needed when electronic systems “go down” or during regular updates.

Care Delivery Considerations

Care delivery might improve if one assumes that machine learning could predict mortality accurately. In cases where patients are predicted to die, clinicians could focus on comfort care, maximizing sedation and pain control. Currently, situations occur when clinicians prioritize other care components, such as hemodynamic stability, to avoid a life-threatening event. For those predicted to live, the perceived objectivity of machine learning could substantiate decisions about using high-risk or resource-, time-, and labor-intensive therapies, such as extracorporeal membrane oxygenation. When decisions involve therapies with high side effect profiles impacting future quality of life, such as an organ or hematopoietic stem cell transplantations, families and clinicians would be better informed to make such choices. One might argue that such models could decrease the decision-making burden on families who are now sometimes asked to contribute to life and death decisions about their child’s care with limited data.^12,13

However, mortality prediction models could also limit the advancement of medical knowledge and family engagement. If clinicians avoid therapies with unknown efficacy for patients predicted to die, we could lose opportunities to learn. If, for example, clinicians in the 1980s stopped resuscitating preterm babies weighing 600 g or less because they had high mortality rates, clinicians would not have learned how to improve survival for these and even smaller infants.¹⁴ If therapies are not pursued based on mortality predictions, emerging data will reinforce current outcomes, stalling progress. Machine learning algorithms can change based on new information (assuming that they are not “locked,” meaning the algorithm does not change with use).¹⁵ However, if mortality predictions are considered immutable and clinical practice reinforces existing outcomes, the data and therefore machine learning algorithms would not change. Further, using such predictors as drivers of care would essentially remove families from the decision-making process, countering current recommendations for family-centered care.¹⁶

Instead of viewing machine learning decisions as immutable, these predictions could provide an adjunct to decision-making. However, how clinicians and patients might use such information is unclear. Clinicians and patients tend to prefer human decision making to that of machines. One study found that radiologists perceived machine learning analysis as lower quality than analysis from humans.¹⁷ Another study showed patients’ reluctance to use health care AI.¹⁷ Education, past experience, user bias, transparency, and inability to account for patients’ uniqueness influence clinician and patient trust in AI.^18,19 These factors are unlikely to change without strong and consistent evidence presented over time supporting the value and accuracy of machine learning predictions. Relying on technology patients find untrustworthy could further erode trust and negatively impact the patient-clinician relationship.

Organizational Considerations

Healthcare organizations may be interested in the financial impact of using machine learning. The initial cost for hospitals purchasing AI technology ranges from $75 000 to $120 000.^20,21 Additionally, the Centers for Medicare & Medicaid Services pays a bonus of about $1000 per patient for institutions that adopt AI systems.²² A substantial financial investment, coupled with the sunk-cost fallacy (the belief that once you have spent money on something, if you spend more money, eventually the thing will work), and government incentives, means that enticements encourage health institutions to adopt AI solutions regardless of their impact on clinician-patient trust or the role of the physician in patient care.

The technological imperative (the belief that new technology is always better) and the economics create situations where physicians might find themselves competing against the AI. For example, most AI is currently used as decision support tools, helping physicians make choices about patient treatment. However, if AI-based recommendations reduced costs, an institution might ask physicians to justify when their treatment decision diverges from AI recommendations. AI might become the standard of care, and physicians rated on “how closely their treatment aligns with the AI.”

For organizations, AI is also a source of revenue. AI companies want healthcare data because large datasets enable more sophisticated and potentially accurate machine learning models.²³ This practice converts patient data into a commodity. Patients become the products, in addition to care seekers. A hospital may want to consider a policy of not sharing pediatric patient data at all to minimize the potential future harms from the loss of privacy. However, that would mean giving up a source of revenue and potentially limiting further development of AI in pediatric care.

Alternatively, pediatric programs could adopt ethical guidelines for sharing and selling pediatric data. For instance, parents could be told how their child’s data are being used. Autonomy suggests that parents should be consented when their child’s data are used, and if the child has the appropriate developmental capacity to understand and reason, the child should provide assent. However, the often emotionally charged, and fast paced PICU setting makes consent conversations challenging and sometimes of limited value. Further, parents have a unique emotional attachment to their child, which could impact their decision-making in currently unknown ways. Minimally, parents should know about such use and preferably be able to opt out of having their child’s data used for AI or sold to other entities. Organizations might also consider policies allowing people to opt out once they reach the age of majority, although the practical implications of such an opt out are significant and potentially prohibitive. Of course, an opt out may lead to systematic omission of data from some populations potentially creating biases in the dataset. Alternatively, an opt out might enhance trust and over-come historic barriers to some populations’ willingness to share their information. Admittedly, extensive data sharing of Health Insurance Portability and Accountability Act (HIPAA) de-identified data, which can be shared without restrictions for commercial and research purposes, already occurs, making it difficult to operationalize these ideals.^24,25

Despite the importance of transparency, hospitals currently use many prediction models as part of “quality improvement efforts” without necessarily disclosing their use to patients.²⁶ This practice reflects the blurry line between hospital operations and medical research. A discussion among stakeholders, including clinicians, ethicists, hospital providers, parents, and patients (including adolescents) is overdue about the moral duties around the use of prediction models. These discussions should at a minimum consider whether to tell parents and patients about the use of prediction models, what should be shared about the use of such models, and how that information should be conveyed.

Legal Considerations

AI also introduces liability questions. Would clinicians be responsible for outcomes that do not match prediction models? Under current law, physicians may be liable for harm to patients if they follow AI recommendations to use nonstandard approaches to care delivery.²⁷ Current law likely only shields physicians from liability when they follow the standard of care. However, if AI becomes part of the standard of care, physicians will likely avoid liability when following (even incorrect) AI recommendations and patient harm occurs.²⁷ This legal space is quickly developing, and it seems that the use of AI might already be nearer to the standard of care than most people realize.^28,29 Even under current law, hospitals may face liability, such as for failing to provide support, training, equipment, or maintenance for an AI system.³⁰ Courts have so far been reluctant to apply product liability, considering physicians as ships’ captains—the final decision-makers.²⁵ In light of the increasing complexity of AI, a more balanced liability system is needed.^30,31

The world of big data presents other legal challenges. The HIPAA Privacy Rule protects only certain health information held by certain designated entities. A vast amount of health-related data are collected by health apps, fitness trackers, and other products outside of the traditional clinical setting where HIPAA applies. Such information may be part of the medical record if held by a hospital or a physician’s office. However, if a technology company collects the data outside of the clinical setting, they have no regulatory responsibility for maintaining patient privacy under HIPAA in most cases, only an ethical duty. Thus, much of individuals’ health privacy data may be left unprotected in the US.^25,32 Increasingly, states (eg, California and Virginia) are adopting more comprehensive privacy protections to give consumers rights over their personal information.^33–35 Collecting data from children introduces more complexities. A child born today will likely already have a “data footprint/identity” when they become old enough to make their own decisions, often without having had any chance to prevent this from happening. Work is needed to understand better when and how to engage children in questions about data sharing, and whether people should be able to retract their data once they reach the age of majority. Data collection in the PICU creates additional challenges because HIPAA protected health information is collected from children who often lack the capacity to consent and thus cannot provide any kind of assent. However, the use of such data for developing AI may bring significant benefits to future pediatric patients, thus arguably making it questionable not to use children’s information.

Conclusions

Although a potentially promising application, there are a vast array of technical, clinical, organizational, and legal concerns that impact the ethics of using machine learning to predict mortality in the PICU. The challenges begin before the development of AI, with the limits of existing healthcare data, and extend through design, implementation, and regulation. Issues related to bias, trust, transparency, liability, and autonomy, among others, could impact the clinician-patient relationship and organizational operations and motivations, and will likely be shaped by a quickly changing, though still underdeveloped, legal landscape. Pediatric patients differ from adults in this landscape because children generally do not have legal control over their data or legal authority to give or withhold consent. Because data can be tracked across a longer proportion of their lives, the implications for privacy harms extend through the lifespan. Considering these issues, greater involvement of multidisciplinary teams of clinicians, computer scientists, parents, patients (including adolescents), ethicists, legal experts, and community members are needed to discuss and shape the application of these promising, powerful, but risky new tools.