Developing machine learning-based models to help identify child abuse and neglect: key ethical challenges and recommended solutions

Abstract

Child abuse and neglect are public health issues impacting communities throughout the United States. The broad adoption of electronic health records (EHR) in health care supports the development of machine learning–based models to help identify child abuse and neglect. Employing EHR data for child abuse and neglect detection raises several critical ethical considerations. This article applied a phenomenological approach to discuss and provide recommendations for key ethical issues related to machine learning–based risk models development and evaluation: (1) biases in the data; (2) clinical documentation system design issues; (3) lack of centralized evidence base for child abuse and neglect; (4) lack of “gold standard “in assessment and diagnosis of child abuse and neglect; (5) challenges in evaluation of risk prediction performance; (6) challenges in testing predictive models in practice; and (7) challenges in presentation of machine learning–based prediction to clinicians and patients. We provide recommended solutions to each of the 7 ethical challenges and identify several areas for further policy and research.

INTRODUCTION

Child abuse and neglect are public health issues that have reached epidemic proportions with more than 700 000 children abused every year in the United States.¹ Child abuse and neglect is defined as any action (physical, emotional, and/or sexual) by a caregiver that results in harm, potential harm, or threat of harm to a child.² There are limited prognostic tools to assess child abuse and neglect in clinical practice³ and health professionals frequently do not detect potential cases of abuse or neglect.⁴ Clinician racial bias has been shown to contribute to overreporting of abuse directed at communities of color.⁵ More than 45% of the children with confirmed abuse and neglect reports were Black or Latinx in 2019 in the United States.⁶

Widespread adoption of electronic health records (EHR) can support the development of clinical decision support systems for identifying child abuse and neglect, thus offering unique insights regarding this phenomenon.^7,8 Our team is conducting an exploratory study aimed at developing a machine learning–based risk model to identify potential child abuse and neglect and reduce racial bias in reporting by processing pediatric emergency department (ED) EHR data. Utilizing EHR data for child abuse and neglect detection raises several critical ethical considerations.⁹ This article discusses key ethical issues related to machine learning–based risk model development and evaluation that emerged during our exploratory study.

We base our analysis on the emerging recommendations for ethical healthcare machine learning development¹⁰ and frame our work in phenomenological ethics, which encourages interdisciplinary research¹¹ according to the principles of self-responsibility in the co-constitutive relationship between the new technology and its users.^12–14 Some of the key questions we pose: is it possible to create an ethically responsible machine learning–based risk model that will prioritize the well-being of children, their families, and the clinicians working with them? Furthermore, how can we adjust the machine learning–based risk model to account for known racial and other user biases?

To provide insights related to these questions, we used a phenomenological ethical lens to: (1) identify and describe key issues in development of a machine learning–based risk model and (2) provide a set of recommendations to address these ethical challenges.

KEY ISSUES IN MACHINE LEARNING–BASED MODEL DEVELOPMENT WITH ETHICAL IMPLICATIONS

When working with machine learning–based risk models we face 2 main types of ethical concerns: epistemic (do I actually know what I am doing?) and normative (what normative direction is the algorithm taking?).¹⁵ Both epistemic and normative concerns were used to identify the 7 key issues described below under 2 general areas.

Area 1—Practicality of machine learning–based risk models: limitations in the available data

Key issue (1)—Biases in the data: Different types of biases based on care giver’s profession, address, educational level, and socioeconomic status might affect the likelihood of suspicion and report of child abuse and neglect.¹⁶ Specifically, there is an emerging body of evidence that racial biases contribute to reporting of child abuse and neglect^17,18 and addressing it in the healthcare system.¹⁹ Because current reporting trends are skewed towards racial minorities, existing data extracted from secondary databases will likely include inherent racial biases.²⁰ For example, using data from the ED EHR might result in an overestimation of child abuse and neglect cases among Black and Latinx families. This overestimation hinders our ability to create unbiased machine learning–based risk models using secondary data.

Key issue (2)—Clinical documentation system design issues: EHRs were originally designed to support healthcare billing and clinical workflows.²¹ As a result, contemporary hospital EHRs have diverse documentation templates, structured and unstructured, to document health and social factors related to child abuse and neglect.^22,23 For example, clinicians often use unstructured documentation (eg, social work and physician notes) to indicate suspicion of child abuse and neglect.

Most medical documentation of child abuse and neglect is in unstructured narrative clinical notes, thus requiring natural language processing to identify the suspected cases and potential risk factors. There is little standardization in collected data, and there are different documentation policies adopted across healthcare settings. This affects our ability to create generalizable and easily shareable machine learning–based risk models that might work in various health settings.²⁴

Key issue (3)—Lack of centralized evidence base for child abuse and neglect: Although child abuse and neglect are widespread in society, little systematic nationwide evidence exists about its risk factors.²⁵ Child Protective Services—state entities charged with addressing child abuse and neglect—are not required to report their data publicly. Our knowledge of risk factors is mostly based on past studies conducted via expert consensus.^26,27 This limited evidence influences our ability to identify a prespecified list of risk factors potentially associated with child abuse and neglect.

Key issue (4)—Lack of “gold standard” in assessment and diagnosis of child abuse and neglect: Currently available national data compiled by the Children Defense Fund indicate vast disparities in the rates of reported child maltreatment cases between states; for example, Kentucky’s average rate per 1000 children is 23.5 cases versus Pennsylvania’s average rate of 1.8 cases.²⁸ This more than 10-fold difference of reported rates in child abuse and neglect is likely not a true difference, but rather a reflection of contrasts in definitions, prevailing culture, child welfare resources, and reporting practices. We also anticipate large differences in numbers of reported cases when EHR data from individual or groups of hospitals are extracted. Coupled with previous issues (key issues 1–3), this lack of concrete outcomes data greatly affects our ability to build “gold standard” datasets that can serve as ground truth for the development of high-quality machine learning–based risk models.

Area 2—Ethics of risk models: machine learning creation challenges

Key issue (5)—Challenges in evaluation of risk prediction performance: Machine learning–based risk models are often evaluated by using predictive performance that includes precision, recall, and other related metrics. Even if a risk model with 99% precision is developed, it is still going to err in 1% of cases. In reality, many clinical machine learning–based risk models do not achieve such high-risk predictive performance; for example, in a recent study for developing a child abuse and neglect machine learning–based risk model, only 16% of cases identified as at risk for abuse were reported by clinicians to child protective services.²³ In the domain of child abuse and neglect, these machine learning–based risk model errors can potentially introduce severe social and emotional consequences for the families who are found to be suspicious for abuse and neglect.²⁹

Further, traditional machine learning–based risk model performance evaluation metrics (eg, precision, recall, and especially F-score) presume that false positives and false negatives are equally undesirable. This assumption of equality of trade-off has not been established or validated for the domain of child abuse and neglect.

Key issue (6)—Challenges in testing predictive models in practice: Conducting a randomized controlled trial is often referred to as a “gold standard” of medical evidence generation. However, conducting a randomized controlled trial of child abuse and neglect machine learning–based risk models may be unethical; in the control group where the risk prediction model would not be applied, there is a risk of missing potential child abuse and neglect cases that might have been identified by machine learning–based risk models. The consequences of missing a potential case are very severe as well as wrongly identifying a case. This affects our ability to understand the impact of the implemented predictive models.

Key issue (7)—Challenges in presentation of machine learning–based prediction to clinicians and patients: Some machine learning–based models (eg, logistics regression or decision trees) are explainable as they produce a list of risk factors associated with negative outcomes. Other models (eg, neural networks) produce predictions of risk with limited ability to identify specific risk factors associated with negative outcomes. Our preliminary results of qualitative interviews with clinicians²⁴ suggest that suspicion of child abuse and neglect is often documented by clinicians in an indirect manner, via specific and sometimes vague language and can be extracted via certain test-orders patterns (eg, orders for certain X-rays). Moreover, machine learning models can base their risk predictions on variables that do not have any intuitive causal relationship to the outcome. As a result, some predictive risk factors might not be easily explainable (or even make clinical sense) and presentable to the clinicians and patients who will need further clarification of the results of the machine learning–based risk models.

Beyond ethical issues, there will be cases where clinicians will either neglect the risk prediction or will proceed contrary to high-risk or low-risk prediction. In those cases, mandatory reporting of maltreatment laws may leave clinicians who overlook a prediction of high risk in legal jeopardy, particularly if a patient comes to harm. This scenario represents new sets of ethico-legal concerns that will need to be addressed when integrating machine learning–based risk predictions into clinical settings.

DISCUSSION

Considering the key issues identified above, stakeholders involved in the machine learning–based risk model development, testing, and implementation need to be aware and responsible for this new technology’s impact. Below, we provide a set of recommendations that can assist in developing such models.

Practicality if creating machine learning–based models

1. Racial biases in the data: Ensuring the diversity of data used to train machine learning–based risk models is central to our ability to decrease machine learning–based risk model biases. However, simply providing diverse training data would not eliminate racial biases. We recommend that researchers design machine learning–based risk models in collaboration with individuals representing diverse lived experiences³⁰ and with different groups of stakeholders, including clinicians, patients, and guardians.³¹ For example, simulation-based co-creation of models can encourage responsible designing of the prediction for all the parties involved by creating user stories and personas, modeling assumptions, and discussing simulated effects.³² Second, artificial intelligence researchers should be thoughtful about applying analytical methods adjusted for biased data, for example over and under sampling³³ and scoring of the module penalization.³⁴ Third, racial and ethnic concordance between patients and clinicians is an important consideration in documentation. Past research has shown that racial concordance can improve communication and information exchange.³⁵ Fourth, stigmatizing language in medical documentation is also a potential source of bias,³⁶ and because of interdisciplinary (physicians, nurses, social workers) differences in documentation styles within EHR,³⁷ researchers should develop tools that provide real-time clinical feedback and recommendations regarding documentation styles and clinical decisions concerning child abuse and neglect. For example, one study flagged unapproved medication abbreviations in clinical notes, which resulted in improving the clarity of clinical documentation.³⁸ Similarly, “flagging” racially biased or otherwise biased language can help to address some of the racial biases. Finally, involving patients in the inputting of demographic data can help to ensure an accurate reflection of their personal identities.³⁹

2. Clinical documentation system design issues: To improve the efficiency of child abuse and neglect assessment, we recommend involving multiple stakeholders such as caregivers, social workers, and patient advocates to assist in designing EHR. The involvement of stakeholders can help improve workflows around medical documentation and redesign how/when data are collected in the ED.⁴⁰ Furthermore, more standardization in the data elements is necessary for child abuse and neglect risk assessment. Additionally, we recommend using semi-automated tools, like natural language processing, to identify child abuse and neglect risk factors. These applications can help reduce some of the barriers associated with documentation issues.

3. Lack of centralized evidence base for child abuse and neglect: More qualitative and quantitative research, beyond expert consensus, is needed to understand the risk factors for child abuse and neglect. This exploration should be done thoroughly to avoid the impact of current biases and differences of child abuse and neglect definitions.

4. Lack of “gold standard” in assessment and diagnosis of child abuse and neglect: Healthcare systems (including hospitals and primary care) and Child Protective Services across the United States should establish safe and reliable data sharing pathways and develop standardized definitions of child abuse and neglect.⁴¹ Insights based on these data are crucial for understanding current child abuse and neglect rates, describing the risk factors, and analyzing the potential racial disparities in reporting. Further, more efforts should aim to create potential linkages between diverse data sources. For example, clinical data from hospitals need to be linked with outpatient social service data and data from Child Protective Services; such linkages will potentially allow the generation of “gold standard” datasets from which high-quality machine learning–based risk models can be developed. These linkages between data sources need to be implemented in a responsible and ethical manner so they are not used to further propagate biases (eg, racial) and enable optimized targeting of certain populations.

Evaluation of machine learning–based risk models

5. Challenges in evaluation of risk prediction performance: To reduce risk prediction error, the evaluation of machine learning–based risk models requires rigorous studies that should be valid and reproducible across diverse populations and over time.³¹ Machine learning–based risk model evaluation requires developing international standards similar to the STROBE guidelines.⁴² There are multiple definitions, measures, and metrics of incorporating fairness into creation and evaluation of machine learning–based models. To evaluate fairness in developing a machine learning–based risk model in child abuse and neglect, one can potentially use an approach that compares groups (eg, racial/ethnic or income groups) by taking into account potential underlying differences between groups; such an approach is also referred to as “confusion matrix-based metrics.”⁴³ Additional fairness approaches should also be examined per definitions provided by Caton and Haas.⁴³ Finally, machine learning–based risk models developers should allow independent evaluation and error analysis throughout the development of the model. This evaluation should be conducted by collaborators that can provide feedback and are not part of the machine learning–based risk model development team.⁴⁴

6. Challenges in testing predictive models in the real world: When randomized clinical trials might not be feasible, an alternative study design, such as an observational or semi-experimental design, might be applied when testing the machine learning–based risk models. In addition, we recommend first validating the model against a training dataset in an internal cross-validation process. Afterward, the machine learning–based risk model should be tested in different clinical settings or different periods from the same setting.⁴⁵

7. Challenges in presentation of prediction results to clinicians: We recommend designing “high-stakes” decision risk models (such as child abuse and neglect machine learning–based risk model) initially as interpretable.⁴⁶ Designing machine learning–based risk models with high interpretability and transparency is essential for providing clinicians and caregivers a clear understanding of how the machine learning–based risk models are working and why decisions are made.⁴⁷ Furthermore, it is vital that results be presented with risk estimates—not as absolute recommendation as the impact of false-positive results can have a long and negative effect on families assessed with child abuse and neglect.

CONCLUSIONS

In agreement with the principles of phenomenological ethics, we encourage responsible use of technology in order to respect the beneficence of the relationship between patients and clinicians, respect the autonomy of the users and achieve social justice for the children, potential victims of neglect and abuse. We believe that clinicians using the new technology still need to be responsible for their final decisions in major decisional steps, such as reporting to child protective services. Even if the machine learning–based risk model makes the work of the clinicians faster and more effective, they will need to remain ethically committed to understanding the relationship between users and technologies^48–50 and see where their role is still crucial to keep a compassionate and just social environment.

FUNDING

This work was supported by the Data Science Institute Seed Funds Program at Columbia University. AL is the recipient of a postdoctoral scholarship from the Haruv Institute.

AUTHOR CONTRIBUTIONS

AL wrote the draft and facilitated group discussion. SF, AB, KC, NA, DP, SS, and MT reviewed and refined the draft and finalized the article.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

No new data were generated or analyzed in support of this research.