Lack of Transparency and Potential Bias in Artificial Intelligence Data Sets and Algorithms

Abstract

IMPORTANCE

Clinical artificial intelligence (AI) algorithms have the potential to improve clinical care, but fair, generalizable algorithms depend on the clinical data on which they are trained and tested.

OBJECTIVE

To assess whether data sets used for training diagnostic AI algorithms addressing skin disease are adequately described and to identify potential sources of bias in these data sets.

DATA SOURCES

In this scoping review, PubMed was used to search for peer-reviewed research articles published between January 1, 2015, and November 1, 2020, with the following paired search terms: deep learning and dermatology, artificial intelligence and dermatology, deep learning and dermatologist, and artificial intelligence and dermatologist.

STUDY SELECTION

Studies that developed or tested an existing deep learning algorithm for triage, diagnosis, or monitoring using clinical or dermoscopic images of skin disease were selected, and the articles were independently reviewed by 2 investigators to verify that they met selection criteria.

CONSENSUS PROCESS

Data set audit criteria were determined by consensus of all authors after reviewing existing literature to highlight data set transparency and sources of bias.

RESULTS

A total of 70 unique studies were included. Among these studies, 1 065 291 images were used to develop or test AI algorithms, of which only 257 372 (24.2%) were publicly available. Only 14 studies (20.0%) included descriptions of patient ethnicity or race in at least 1 data set used. Only 7 studies (10.0%) included any information about skin tone in at least 1 data set used. Thirty-six of the 56 studies developing new AI algorithms for cutaneous malignant neoplasms (64.3%) met the gold standard criteria for disease labeling. Public data sets were cited more often than private data sets, suggesting that public data sets contribute more to new development and benchmarks.

CONCLUSIONS AND RELEVANCE

This scoping review identified 3 issues in data sets that are used to develop and test clinical AI algorithms for skin disease that should be addressed before clinical translation: (1) sparsity of data set characterization and lack of transparency, (2) nonstandard and unverified disease labels, and (3) inability to fully assess patient diversity used for algorithm development and testing.

Artificial intelligence (AI) algorithms have been widely applied to clinical tasks involving images, ranging from prediction of diagnoses using chest radiographs to assessment of skin lesions for malignant disease. Clinical deployment is the ultimate goal for developing AI algorithms in medicine. To be successful, AI algorithms need to be trained and tested on data that represent clinical scenarios encountered in real-world settings.¹ The data that are used to train and test a model can determine its applicability and generalizability.^1,2 Therefore, a clear understanding of data set characteristics (eg, how training data were collected, labeled, and processed) is critical.³ Moreover, a description of data demographics is key to understanding the generalizability of any data set to clinical practice.⁴ The machine learning community has suggested the importance of using standards for describing data sets, building from the idea of standardized data sheets used in the electronics industry, but such standards have not yet been implemented by scientific journals or regulatory authorities.^3,4

For nonmedical deep learning applications, large data sets are publicly available and easily evaluated. However, medical data sets are often siloed owing to concerns about patient privacy.⁵ Because of the inaccessibility to the underlying data used to develop models, data set descriptions are integral to understanding the generalizability, or lack thereof, of a described model. Within the medical community, no broad standards exist at present that are universally applied for describing data sets used in developing AI models. Data set descriptions for specific applications often do not address potential sources of bias.^6,7 Most US Food and Drug Administration–approved medical AI devices provide little public information on what data and demographic groups were used for testing.² Guidelines such as Consolidated Standards of Reporting Trials–AI and Standard Protocol Items: Recommendations for Interventional Trials–AI have addressed transparency in AI clinical trials but do not cover initial AI development and early testing.^7,8 Thus, there have been calls for increased transparency in medical AI.⁹ To better understand the characteristics of data sets used for developing AI algorithms in dermatology, we performed a scoping review that assessed data set availability and data set descriptions such as image labeling and patient diversity. We used this information to assess potential pitfalls in dermatology AI data sets.

Methods

We selected a scoping review approach to allow us to identify the data sets used in existing AI development, to assess data set transparency, and to extract the relevant features that may contribute to bias. We searched PubMed for peer-reviewed articles published between January 1, 2015, and November 1, 2020, that defined a clinically relevant task and developed a deep learning algorithm or that tested an existing deep learning algorithm using clinical or dermoscopic images of skin disease. We constrained the search to English language papers. We used the following pairs of search terms: deep learning and dermatology, artificial intelligence and dermatology, deep learning and dermatologist, and artificial intelligence and dermatologist. We excluded published reviews. The methods and results sections were thoroughly reviewed by one investigator (R.D.) to select studies that met criteria, while another investigator (M.P.S.) independently reviewed the selected abstracts to confirm that they met selection criteria.

Data set audit criteria were determined by consensus of all authors after reviewing existing literature to highlight data set transparency and sources of bias; extracted features are shown in the eTable in the Supplement.^4,6 An audit was performed by 2 investigators (R.D. and M.P.S.), with each investigator independently reviewing the full study and supplemental data and reconciling any differences via consensus. Another investigator (M.D.S.) performed a final review to ensure the accuracy of the extracted data. For each data set, we analyzed the sources, the number of images, the number of patients involved, how the data set was used (for training, internal validation, testing, or external validation), whether the data set was clinical or dermoscopic, the diseases studied, how images were labeled and whether the labeling met the gold standard criteria (defined as pathological findings for malignant neoplasms [eMethods in the Supplement]), the descriptions of Fitzpatrick skin type and race/ethnicity, whether the data were publicly available or private, whether image processing was described, and whether the deep learning model was available (eTable in the Supplement).

Data set descriptions were based on how data were presented in the study. For example, if the original study distinguished between different data sets from separate sources in its analysis, then these data sets would also be separated in our audit. Multisource data sets that were presented as combined without indication of how many images came from each source in the original study were considered as a single data set for the audit. Images scraped from internet searches, rather than from defined atlases and repositories, were considered to be private data sets if the images were not shared with their applied labels. Synthetic images were not considered in the image count. To meet criteria for describing data set features in the eTable in the Supplement, the information needed to be included in the main text or supplement, even for data sets that are publicly available. Any discrepancies in reported images are noted in the eTable in the Supplement. The study was not registered with PROSPERO, and a review protocol was not published because such protocols do not apply to scoping reviews.^10,11

Results

We identified 70 studies that met our criteria.^12–81 Of these studies, 57 developed a new deep learning algorithm and 13 tested an existing algorithm on an additional data set (Table 1 and eTable in the Supplement). Most of the studies were published within the last 2 years (eTable in the Supplement). Among the 57 studies that trained and tested a new algorithm, only 14 had test data that were from a different source than the training set or an additional external validation. For studies training and testing a new deep learning algorithm, the mean (SD) total number of images was 23 222 (47 061) and the median was 10 015 (interquartile range, 2200–14 016), whereas the number of images used in studies testing an existing algorithm had a mean (SD) of 682 (894) and a median of 212 (interquartile range, 100–780).

Table 1.

Summary of Key Findings

Description	No./total No. of studies (%)
Developed a novel algorithm	57/70 (81.4)
With novel algorithms that tested on an external test set	14/57 (24.6)
With tasks involving cutaneous malignant neoplasms	56/70 (80.0)
Involved cutaneous malignant neoplasms and reported biopsy-proven labels for all cutaneous malignant neoplasms	36/56 (64.3)
Reported any race or ethnicity information for ≥1 data set	14/70 (20.0)
Reported any Fitzpatrick skin tone information for ≥1 data set	7/70 (10.0)
Stated that the AI model developed or used was publicly accessible	21/70 (30.0)

Abbreviation: AI, artificial intelligence.

In terms of diseases, 56 studies included at least 1 cutaneous malignant neoplasm in their task (eTable in the Supplement). The gold standard for cutaneous malignant neoplasm is histopathological diagnosis.¹ Of the 56 studies with a cutaneous malignant neoplasm diagnosis as part of the task, 36 (64.3%) met gold standard criteria, meaning that every cutaneous malignant neoplasm in the data set was confirmed by pathological findings (eTable in the Supplement). This suggests that a substantial number of studies were trained and/or tested on noisy annotations, which could affect diagnostic accuracy in clinical practice.

Among all studies, only 14 of 70 (20.0%) described any information on the ethnicity or race of patients in at least 1 data set and only 7 of 70 (10.0%) included any information on the Fitzpatrick skin types of images used in at least 1 data set (eTable in the Supplement). Studies that used public data sets with previously described demographic information (see Table 2) did not always include this information. Fitzpatrick skin typing represents the skin’s response to UV radiation, and dermatologists often use it as a proxy for skin color.⁸² Of the 7 studies with Fitzpatrick skin type information, 4 reported having no images of the darkest skin types (Fitzpatrick skin types V and VI) (eTable in the Supplement). We assessed the 3 studies^12–14 that included Fitzpatrick skin types V and VI and found the information to be either incomplete or to underrepresent darker skin tones. Dulmage et al¹² reported skin tone for just a subset of the test data and grouped Fitzpatrick skin types IV to VI images together, making it impossible to assess how many individual Fitzpatrick skin type V and VI images were included. Liu et al¹³ had 3.2% type V skin images and 0.3% type VI skin images in the training data and 2.7% type V skin images and 1 type VI skin image in the test data. Phillips et al¹⁴ tested an existing algorithm on an external test set that included 2.0% type V skin images and 0.8% type VI skin images. Data on the prevalence of Fitzpatrick skin types in the US population are inadequate. One study⁸³ reported that the general US population includes 3.6% type V skin and 9% type VI skin, although this could be an underrepresentation, because the percentage of racial and ethnic minority group patients included in this study was lower than the reported demographics of the US population. Even where Fitzpatrick skin type was annotated and reported,^84,85 the authors did not evaluate the algorithm in person, but rather reviewed post hoc images without information about lighting conditions, which may lead to inaccuracies and additional noise.

Table 2.

Public Data Sets Used in 3 or More Publications

Data set source	No. of images	No. of patients	Type of image	Diseases	Label	Gold standard for malignant neoplasm diagnosis (pathological finding)	Fitzpatrick skin type description and breakdown	Ethnicity description and breakdown
ISIC 2016 Challenge	1279	Not specified	Dermoscopic	Nonmelanoma and melanoma	Benign (nonmelanoma): expert consensus; malignant (melanoma): pathological findings	Yes	No	No
ISIC 2017 Challenge	2750	Not specified	Dermoscopic	Benign nevi, seborrheic keratosis, and melanoma	Benign nevi: expert consensus; seborrheic keratosis: expert consensus; melanoma: pathological findings	Yes	No	No
ISIC 2018 Challenge (HAM10000)	10 015	Not specified	Dermoscopic	Actinic keratosis, intraepithelial carcinoma (Bowen disease), BCC, benign keratosis, dermatofibroma, melanocytic nevi, vascular skin lesions, and melanoma	Actinic keratosis: consensus; intraepithelial carcinoma: pathological findings; BCC: pathological findings; benign keratosis: consensus; dermatofibroma: consensus; melanocytic nevi: consensus; vascular skin lesions: consensus; melanoma: consensus	Yes	No	Yes; nationality breakdown (as a percentage of the 10 015 images in the data set): 2.0% Portuguese (PH2); 22.6% Australian (Rosendahl); Austrian (ViDIR) not specified (Atlas and ISIC 2017)
Hellenic Dermatological Atlas	2663 (as of April 2021)	Not specified	Clinical	Various: 43 broad categories of disease	Not specified	Unable to assess	No	No
D@nderm Atlas of Clinical Dermatology	>3000 (as of April 2021)	Not specified	Clinical	Various: Common skin diseases under 9 broad categories	Not specified	Unable to assess	No	No
MED-NODE database	170	Not specified	Clinical	Melanoma and nevi	Nevi: pathological findings; melanoma: pathological findings	Yes	No	No
Edinburgh Dermofit Library	1300	Not specified	Clinical	Actinic keratosis, BCC, melanocytic nevus (mole), seborrheic keratosis, SCC, intraepithelial carcinoma, pyogenic granuloma, hemangioma, dermatofibroma, and malignant melanoma	Expert opinion (including dermatologists and dermatopathologists) based on clinical information and pathological findings	Yes	No	No
DermNet NZ	>20 000	Not specified	Clinical	Various: 1000s of categories listed	Not specified	Unable to assess	No	No

Abbreviations: BCC, basal cell carcinoma; ISIC, International Skin Imaging Collaboration; SCC, squamous cell carcinoma; ViDIR, Vienna Dermatologic Imaging Research.

To the extent that it was possible given the lack of clear labeling in many studies, we estimate that of a total 1 065 291 images, 257 372 (24.2%) were publicly available or available on request, whereas 807 919 (75.8%) were private. The accuracies of these estimates are affected by the fluctuation of available images in public databases and the inability to evaluate the presence of overlap among private databases. For example, some groups published multiple studies but did not indicate how much overlap there was between data sets from the same hospital system or academic center,^15,16 whereas some groups did not even specify the source of their data.¹⁷ Studies that used publicly available data sets did not always describe what subsets of the data sets they used or reference previously described demographic information or labeling information.^22,30,55 In some cases, multisource data sets were created with both private and public data but without indicating the number of images or diagnoses contributed by each source.¹⁸

Among the publicly available data sets, the International Skin Imaging Collaboration repository was used in the largest number of studies (n = 28), followed by MED-NODE: University Medical Center Groningen (n = 5), and Hellenic Dermatological Atlas (n = 4).^86,87 The features of the publicly available data sets used by 3 or more studies are described in Table 2.^{19,20,86–92} Most of the AI literature addressing skin conditions is developed from private data sets used once to generate a single study, with no ability for replication (Figure), which greatly limits their effect and generalizability. Public data sets are drivers of model development and publication while allowing greater transparency (Figure). Across all the studies, models were generally not accessible, with only 21 of the 70 studies (30.0%) stating that the model was available on request, publicly, or through a commercial product (eTable in the Supplement).

Figure.

Overview of Data Sets and Studies. Squares represent studies; circles, data sets; and arrows, use of a data set. The number of images in a given data set is represented by the size of the circle. Private data sets are often only connected to 1 study, whereas public data sets help generate multiple studies. A mapping of the corresponding data sets and studies is provided in the eFigure in the Supplement.

Discussion

Developing clinically applicable deep learning models in medicine is predicated on the creation of robust models developed from data sets that are either publicly available for scrutiny or well described. Clinical AI must be held to a high standard owing to the potential for significant medical harm from errors and bias.² Previous studies⁹³ have shown that direct-to-consumer dermatology AI products have significant performance issues. Algorithmic performance depends significantly on the data on which it is trained and tested. Given the significant effort and cost that it requires to generate a data set for algorithmic development, we provide recommendations that should be implemented prospectively when data sets are being collated and algorithms are being developed. Using skin disease as a case study, we analyzed 70 studies related to AI and dermatology to understand the state of data set transparency and sources for potential bias.

We found that most data used for training and testing AI algorithms in dermatology were not publicly available. Most studies using private data did not describe important demographic information such as ethnicity/race or skin tone. Most developed models were likewise not available for additional evaluation to assess robustness. The lack of details and transparency about data and models limits our ability to systematically assess robustness and potential biases, which is an important direction of future work.

Based on our findings, we provide 3 recommendations for improving transparency and reducing the potential for bias in developed clinical algorithms. First, we recommend that whenever feasible, data sets and/or models be shared at time of publication. Even if the full model cannot be made public owing to intellectual property issues, developers should at least provide application programming interface access to the model to researchers using platforms such as gradio.⁹⁴ If images are from public data sets or repositories, the images that were used should be clearly delineated. Images scraped from the internet via web search should be shared with their attached labels; these data cannot be considered publicly available if it is impossible to identify which images were used or how they were labeled. Owing to issues around patient privacy, medical data sets are not always readily available; however, repositories such as International Skin Imaging Collaboration have shown how deidentified data cannot only be shared but also catalyze algorithmic development (Figure). Direct access to data sets allows direct assessment of any potential biases. For example, data in International Skin Imaging Collaboration do not include skin tone information, but external groups have attempted to improve labeling for skin tone diversity through their own analyses.⁸⁵

Second, if data sets cannot be shared, there should be a clear description of important data set characteristics, which include patient populations, skin tones represented, and the image labeling process.⁴ If available, other metrics that should be reported include data set sources (such as hospital or clinic), cameras used for photograph capture, data set processing steps, and information on photograph quality, because these have also been shown to affect AI performance.²¹ The aforementioned characteristics are not exhaustive; in fact, a challenge for this study was the lack of dermatology-specific guidelines for defining and describing AI data set characteristics. Through this scoping review, we found that few studies mention patient race or ethnicity race or patient skin tone. Previously, dermatologists have expressed concern about the potential of AI algorithms to perpetuate disparities by the exclusion of diverse skin tones,⁹⁵ and this effect has been shown in an example where performance significantly differed between lesions on White vs Asian patients.¹⁶ However, assessing racial and ethnic diversity among patients in these data sets was limited by sparse reporting of these characteristics. The few studies that include skin tone information have underrepresented darker skin tones when compared with the general US population. Many AI algorithms aim to be used globally, and these algorithms should be trained and tested on data that represent global diversity.

Data label noise is another potential source of algorithm vulnerability. In the machine learning literature,⁹⁶ mislabeling can lead to algorithms that are good predictors of the mislabeled data but poor predictors of the ground truth. For clinical AI, it is imperative that studies clearly delineate how labels were generated and whether these labels constitute the gold standard for diagnosing that disease. Multiple dermatology AI studies^13,18 have relied on consensus panels for the diagnosis of cutaneous malignant neoplasms, such as melanoma. However, studies⁹⁷ have shown that the number needed to biopsy among dermatologists to identify 1 melanoma is 7.5. This means that a significant portion of images labeled melanoma by a consensus panel without histopathological confirmation may not actually be melanoma.

Third, there should be a clear description of how data sets were used, that is, for training, validation, testing, or additional external validation. Running models on external data sets is an important step for demonstrating algorithmic robustness. Previous studies^2,21 have demonstrated that significant performance drops can occur when algorithms trained exclusively at a single site are applied to an external site.

Limitations

Limitations of this study include the sparsity of reported data, which hampered our ability to fully assess some of the factors we were interested in assessing, such as skin tone and racial and ethnic diversity of the data sets. In addition, we focused on PubMed indexed articles; however, the AI literature has many preprints and conference proceedings that may not appear on PubMed. The studies featured in our scoping review are a representative sample of the AI in dermatology literature.

Conclusions

As of early 2021, no US Food and Drug Administration–approved AI devices have addressed cutaneous diseases, yet innovation is occurring at a rapid pace. This scoping review of the data sets used for developing AI algorithms for cutaneous disease reveals concerns about the lack of transparency and inadequate reporting of important data set characteristics, including racial and ethnic diversity among patients. At present, no reporting guidelines are available for AI data sets used to develop algorithms in dermatology. Specific dermatology AI development guidelines could improve data set transparency and the ability to assess bias in AI applications for dermatology.

Key Points.

Question

How transparent are the data sets used to develop artificial intelligence (AI) algorithms in dermatology, and what potential pitfalls exist in the data?

Findings

In this scoping review of 70 studies addressing the intersection of dermatology and AI that were published between January 1, 2015, and November 1, 2020, most data set descriptions were inadequate for analysis and replication, disease labels did not meet the gold standard, and information on patient skin tone and race or ethnicity was often not reported. In addition, most data sets and models have not been shared publicly.

Meaning

These findings suggest that the applicability and generalizability of AI algorithms rely on high-quality training and testing data sets; the sparsity of data set descriptions, lack of data set and model transparency, inconsistency in disease labels, and lack of reporting on patient diversity present concerns for the clinical translation of these algorithms.