Invited commentary: deep learning—methods to amplify epidemiologic data collection and analyses

Abstract

Deep learning is a subfield of artificial intelligence and machine learning, based mostly on neural networks and often combined with attention algorithms, that has been used to detect and identify objects in text, audio, images, and video. Serghiou and Rough (Am J Epidemiol. 2023;192(11):1904-1916) presented a primer for epidemiologists on deep learning models. These models provide substantial opportunities for epidemiologists to expand and amplify their research in both data collection and analyses by increasing the geographic reach of studies, including more research subjects, and working with large or high-dimensional data. The tools for implementing deep learning methods are not as straightforward or ubiquitous for epidemiologists as traditional regression methods found in standard statistical software, but there are exciting opportunities for interdisciplinary collaboration with deep learning experts, just as epidemiologists have with statisticians, health care providers, urban planners, and other professionals. Despite the novelty of these methods, epidemiologic principles of assessing bias, study design, interpretation, and others still apply when implementing deep learning methods or assessing the findings of studies that have used them.

This article is linked to “Deep Learning for Epidemiologists: An Introduction to Neural Networks” (https://doi.org/10.1093/aje/kwad107).

Editor’s note: The opinions expressed in this article are those of the authors and do not necessarily reflect the views of the American Journal of Epidemiology.

 

Artificial intelligence via deep learning models underlies many of our daily technological tools and online social interactions, encompassing activities such as object identification in images and videos, speech recognition, text analysis and search, among many others, and is beginning to enter the biomedical and public health research arenas.^1-6 Deep learning has also been used for identifying health conditions from radiological images and genomic data,^7-12 extracting data from clinical charts and notes,^13-15 identifying racist content on social media,^16,17 building chatbots to deliver health information,^18,19 and classifying the built and social environments.^20-26 In this issue, Serghiou and Rough²⁷ provide a timely primer on the fundamentals of deep learning for epidemiologic researchers, focused on describing the mathematical and statistical basis for these methods. In this commentary, we focus on the role deep learning could take in epidemiology, how deep learning could be useful to epidemiologists, and how epidemiologists should approach these methods.

Deep learning presents a potentially powerful means to speed and expand some traditional data collection methods used in epidemiology by replacing some of the human labor required for extraction from qualitative data such as patient charts,¹⁴ free responses from surveys and interviews,²⁸ social media,¹⁷ neighborhood measures from in-person and virtual audits or GIS (geographic information system),^21,25 and audio and video recordings,^29-31 although it also, of course, presents other challenges and issues. Studies using these data-collection methods are frequently limited by the time and effort required for data extraction, as well as the geographic area they can cover.^20,25 For example, suppose a researcher wants to examine the relationship between some built environment feature, like sidewalk conditions, and some health outcome, like older adult falls. In that case, they will need spatially precise measures of sidewalk conditions.^32,33 Obtaining such measures often requires manual collection efforts that can be time-consuming and costly, including examining maps, images, or other sources to identify and quantify the sidewalk conditions. Similar efforts could be expended with deep learning by using the collected data as training data for neural networks, allowing researchers to automatically quantify the presence and condition of sidewalks more efficiently in a much larger geographic area using archived imagery such as Google Street View or Mapillary.²⁵

The potential to leverage such efficiency, as described by Serghiou and Rough,²⁷ relies on the training data from which neural networks can learn to identify or detect the object of interest (eg, sidewalk conditions). These training data are usually developed by trained human raters who identify the objects, features, or words of interest in the raw data. It follows that future-thinking epidemiologists might choose to plan and design present extraction protocols to prepare for the results to be used as training data, either in their current research or for future use. Even if no such steps can be taken, protocols previously relying on human labor could use such labor only to collect training data, allowing deep learning to replace much of the originally planned manual collection, potentially expanding the study to a larger geographic area or population. Just as with any statistical analysis, however, deep learning approaches also require training data, as well as rigorous development of artificial intelligence models through training, testing, and validation that may also require substantial time and costs.

Deep learning can also contribute to or supplement and augment current and traditional statistical modeling approaches and data analyses used by epidemiologists, particularly in cases with large or high-dimensional data.^34,35 Some recent evidence suggests that deep learning models may perform better than traditional models for creating propensity scores,³⁶ prediction for screening and diagnostic instruments,³⁷ and for causal inference in observational studies,^38-40 whereas for multiple imputation there are conflicting findings.^41-44 There are also efforts and possibilities to use deep learning for predicting individual event censoring time,⁴⁵ analyzing count data,^46,47 improving nutritional epidemiology models,⁴⁸ and estimating population prevalence or risk of diseases and mortality.^49,50 Such approaches, essentially considering deep learning as a tool that exploits computational power to wring every bit of information from existing data, are simultaneously exciting and warrant caution regarding the overfitting of those models to their training data.⁵¹ Additionally, as these approaches are still novel and have not been applied extensively in epidemiologic and public health research, the extent of their limitations may not yet be fully understood.

Deep learning may also unlock new opportunities for cross-site data sharing.⁵² For example, deep learning-driven natural language processing and image recognition algorithms could be used with potentially personally identifying electronic health record data, including clinical notes or diagnostic images, to generate nonidentifying data abstracts that could be shared with researchers not approved to see the original, identifying data. In principle, such approaches could dramatically lower the burden of the assembly of pooled datasets, significantly increasing the scope of data collaboratives.^53,54 However, novel approaches to data sharing also raise novel methodological concerns and may deepen or complicate existing concerns. As another example, suppose a natural language processing algorithm that enables cross-site sharing of clinical notes identifies clinical conditions better on sites that use a particular electronic health record system or point-of-care note-taking tool. Such a scenario might induce selection bias (eg, the algorithm is used to identify cases only and not controls) or other systematic measurement error (eg, the algorithm is used to code exposure or other variables of interest). Researchers must be aware of this risk before launching the study to gather enough training data on sites other than the site on which the algorithm was trained if they hope to estimate the error and put bounds on the bias.

As briefly mentioned by Serghiou and Rough,²⁷ there are novel approaches in the field that readers should be aware of, particularly in generative pretrained transformer models (also known as foundation models). Transformer models have been a significant recent advance in artificial intelligence. They have gained substantial public awareness, especially during the past year with publicly available generative text (eg, ChatGPT), image (eg, Midjourney or DALL-E), video (eg, Make-A-Video), programming code (eg, Github Copilot), and other models.^55,56 The transformer model was first described by researchers from Alphabet⁵⁷ for natural language–processing models and later extended to computer vision by researchers at Facebook Artificial Intelligence Research.⁵⁸ Unlike traditional neural networks, transformer models can learn valuable representations in unsupervised scenarios (ie, they do not require the research team to train the model to recognize a particular outcome) but typically require extremely large datasets (eg, billions of data elements or parameters as opposed to millions in the largest traditional neural networks) to outperform traditional neural networks. Training a transformer from scratch, however, is not always necessary (let alone feasible) for researchers interested in using these models; rather, existing trained models can be combined with new data of interest to create models more specific to the research question or task of interest. These models or traditional neural networks can also be enhanced with fine-tuning, such as using pretraining and self-supervised learning.^59,60

How should epidemiologists evaluate evidence from studies using deep learning and what should those articles include? As with all epidemiologic studies, issues of validity, reliability, and bias remain paramount. To allow readers to assess these threats, studies incorporating deep learning should describe the training data, the models used, model parameters, how the model was trained, any fine-tuning steps to improve the models, model performance, and ethical aspects.⁶¹ Describing the training data is extremely important, as much of the bias and limitations from models may originate from them. This should include how the training data were collected (eg, sampling methods used), the reliability and validity of those data (eg, interrater reliability or other quality control statistics), and making training data available, if possible, for others to examine and use. Sharing the models is a very common practice among researchers in deep learning, and if possible, sharing model weights is another consideration so the findings can be replicated and applied to other datasets. Several online communities, such as Github, HuggingFace.co, and Deepai.org, enable researchers to share code, models, and data. Shared, common data resources for the epidemiologic community could help improve the rigor and generalizability of these models as their use grows. Providing open access to the data and models from epidemiologic studies could enable replicability, as well as provide some of the resources needed to make effective use of artificial intelligence approaches. For example, existing benchmark datasets widely used already for deep learning are ImageNet, COCO (common objects in context), PaLM, LLaMA-2, Mapillary Vistas, but few of these may meet the needs of epidemiologic research, particularly for more specialized topics.⁶² The quality of these existing datasets also varies substantially and may not meet the reliability needed for epidemiologic research, especially for complicated or specific tasks, such as detecting and identifying small objects of the built environment, food and nutrition, racism and discrimination, medical images, and other video, audio, or text data relevant to public health and epidemiology. Epidemiologic research could greatly benefit from a set of curated models and training datasets that have been validated, with high reliability.

Just as epidemiologists collaborate with statisticians, health care professionals, and other research professionals, those seeking to use deep learning in their research and practice should seek out partners and collaborators who are experts in deep learning to ensure fidelity and appropriate implementation and interpretation in epidemiologic research. Just as with other disciplines, initial collaborations can be challenging as we seek common language and methods, but are well worth the effort. We can expect many exciting and innovative uses of these methods in epidemiology and public health research and implementation, as well as misuses and ethical challenges, as they become more common and accessible. As with any new methods or technological advances adopted by the field, the underlying epidemiologic principles of interpretation, study design, bias, causality, and others will still apply.

Funding

This work was supported by the Fogarty International Center of the National Institutes of Health (grant K01TW011782 to D.A.Q.); by the National Library of Medicine (grants R00LM012868 to SJM and R01LM012849 to Q.C.N.); and by the National Institute on Minority Health and Health Disparities (grant R01MD016037 to Q.C.N.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflict of interest

We have no conflicts to report.

Data availability

Data was not used for this work.