Use of Deep Neural Networks in the Detection and Automated Classification of Lesions Using Clinical Images in Ophthalmology, Dermatology, and Oral Medicine—A Systematic Review

Abstract

Artificial neural networks (ANN) are artificial intelligence (AI) techniques used in the automated recognition and classification of pathological changes from clinical images in areas such as ophthalmology, dermatology, and oral medicine. The combination of enterprise imaging and AI is gaining notoriety for its potential benefits in healthcare areas such as cardiology, dermatology, ophthalmology, pathology, physiatry, radiation oncology, radiology, and endoscopic. The present study aimed to analyze, through a systematic literature review, the application of performance of ANN and deep learning in the recognition and automated classification of lesions from clinical images, when comparing to the human performance. The PRISMA 2020 approach (Preferred Reporting Items for Systematic Reviews and Meta-analyses) was used by searching four databases of studies that reference the use of IA to define the diagnosis of lesions in ophthalmology, dermatology, and oral medicine areas. A quantitative and qualitative analyses of the articles that met the inclusion criteria were performed. The search yielded the inclusion of 60 studies. It was found that the interest in the topic has increased, especially in the last 3 years. We observed that the performance of IA models is promising, with high accuracy, sensitivity, and specificity, most of them had outcomes equivalent to human comparators. The reproducibility of the performance of models in real-life practice has been reported as a critical point. Study designs and results have been progressively improved. IA resources have the potential to contribute to several areas of health. In the coming years, it is likely to be incorporated into everyday life, contributing to the precision and reducing the time required by the diagnostic process.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10278-023-00775-3.

Introduction

In recent years, the potential of technological resources involving artificial intelligence (AI) to assist in the processes of health diagnoses has been discussed. Its incorporation is based on pattern recognition from the analysis of clinical images [1]. Static images, video clips, and sound multimedia are routinely captured in cardiology, dermatology, ophthalmology, pathology, physiatry, radiological oncology, radiology, endoscopies, and other medical specialties [2].

The increasing interest on the insertion of AI technologies in the health area is based on the fact that many people have limited access to specialized health services, as well as a shortage of specialists working in rural areas or far from large centers [3–15]. It may be assumed that the incorporation of these resources could offer clinical support to health professionals to speed up decision-making regarding diagnosis and management of different diseases [1, 11, 16–18]. The combination of interactive documentation with metadata, image annotations with text, tables, graphics, and hyperlinks optimize communication between medical professionals [2, 19]. The incorporation of these resources with artificial intelligence has been gaining prominence in the health area [2].

Studies have produced promising evidence from the application of AI and deep learning (DL) technologies. The results were equivalent or even superior to those of human experts in the identification of visual patterns and automated classification of clinic images [20, 21], including the classification of ophthalmic diseases [16, 17, 22–25], skin lesions [4, 5, 21, 26–29], and change genetic/phenotypic lesions [30], among others, with high sensitivity and specificity.

The incorporation of AI capabilities in ophthalmology has shown promising impacts in mass screening programs [16, 25]. Diabetic retinopathy (DR) is a common complication of diabetes mellitus and the leading cause of blindness [16], while glaucoma can cause irreversible blindness, with worldwide projections of 111.8 million by 2040 [22]. Early diagnosis is essential and can be done by analyzing fundus photographs [16, 31, 32]. Automated image classification can become an enabler to support screening services in resource-limited environments [16, 25, 31, 32].

The literature demonstrates relevance in the use of artificial intelligence for the early diagnosis of skin diseases that affect 1.9 billion people worldwide [5]. There are an estimated 6480 melanoma diagnoses in the USA in 2019, resulting in 7230 deaths [33]. On the other hand, basal cell carcinoma (BCC) is the most common skin cancer, and although not usually fatal, it imposes a great burden on health services [26, 33]. Due to the shortage of dermatologists, general practitioners see many cases, with less diagnostic accuracy [5]. Deep learning systems demonstrate equivalent performance to specialists and superior to general practitioners, highlighting the great potential to aid in the diagnosis of other skin pathologies [4], such as psoriasis [28, 34], onychomycosis [27], acne [35], and melanoma [36].

Other studies have been conducted with the use of AI and machine learning (ML) in the field of oral medicine, for different purposes, like predict the survival of patients diagnosed with oral cancer to support early diagnosis and definition of the treatment plan. Many studies describe the association of AI with resources, such as computed tomography [37, 38], autofluorescence [39, 40], hyperspectral images [41, 42], intraoral probes [43], and photodynamic therapy [44]. Even if incipient, the great potential of technology [4, 5, 16, 17, 22–28, 45–47] with performance comparable to that of specialists is demonstrated [1, 3, 48].

Therefore, the objective of this review was to perform a systematic review about the performance of artificial neural networks and deep learning in automated recognition and classification of lesions from clinic images in different health fields, comparing them with human performance.

Materials and Methods

The review was conducted using a predetermined protocol which followed the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement and checklist [49]. The protocol was registered in the International Prospective Register of Systematic Review (PROSPERO) database, CRD42021243521.

Review Question and Search Strategy

The research question of the study was: “How effective is the use of deep artificial neural networks in the automated recognition and classification of lesions and pathological changes, from clinical images in the areas of ophthalmology, dermatology and oral medicine when compared to human evaluation?”.

Electronic searches without publication date or language restriction were made in four databases in January 2021 (and updated on January 2022), as the following: PubMed (National Library of Medicine), Web of Science (Thomson Reuters), Scopus (Elsevier), and Embase (Elsevier). Keywords were defined according to Medline (MESH), Embase (Emtree), and Capes (DECs), with combinations of the following terms: “Dermatology,” “skin diseases,” “Ophthalmology,” “eyes diseases,” “oral medicine,” “stomatology,” “artificial intelligence,” “machine learning,” “neural network,” “computational intelligence,” “machine intelligence,” “computer reasoning,” “AI (Artificial Intelligence),” “computer vision systems,” “computer vision system,” “convolutional neural network,” “photography,” “photographies.” In addition to the electronic search, a manual search was performed using references cited in the identified articles and also a search in Gray Literature (Google Scholar and CAPES Bank of Theses).

Eligibility Criteria

Studies evaluating the effectiveness of using deep artificial neural networks to automated classification of pathological changes pertaining to ophthalmology, dermatology, and oral medicine based on their clinical images (photos) were included. There was no restriction by language or year of publication. Reviews, ratings, letters, comments, personal or expert opinions, follow-up studies, and event summaries were excluded.

Study Selection and Data Extraction

Reference management was performed using the EndNote X7.4 software (Clarivate Analytics, Toronto, Canada). Duplicates were removed upon identification. After duplicate removal, two authors (RFTG and EFH) reviewed titles and abstracts of all studies. If the title and abstract met the eligibility criteria, the study was included. A third research group member solved possible disagreements between the two authors (RMF).

The following items were extracted from the articles: name of the author(s) and year of publication; country, health, and pathology area to which it refers; details of the artificial neural network technique used; standardization in image capture; use of public databases for training and validation; pre-processing of images; metrics used to measure performance; main results; and reported limitations.

Assessments of the Risk of Bias and Quality

Risk of bias and study quality analyses were performed independently by two authors (RFTG and EFH). This evaluation was based on the Quality Assessment of Studies of Diagnostic Accuracy (QUADAS) questionnaire adapted [50], where only the items that applied to the study design were considered, whose criteria are depicted in Table 1. The table demonstrating the quality assessment of the studies is included as Supplementary Material 1.

Table 1.

Quality assessment of studies of diagnostic accuracy [50]

Item	Description	Aspects considered in the evaluation	Score quality*
1	Patient representation	This criterion was evaluated considering the importance of a large volume of data for the development and validation of AI models (n = clinical image)	57.6%
2	Clear selection criteria	Evaluation of the division of the database into training (70%) and validation (30%) groups	91.5%
3	Sample receives verification using a reference standard	Considering that the studies were based on the interpretation of a clinical image, and not on the gold standard, to evaluate this aspect, considering the randomization processes of the training and validation bank	45.7%
4	Independent reference standard	In this criterion, it was considered that the index test using AI was not part of the reference standard	100.0%
5	Index test described in detail	We consider the description developments of AI models, architectures, equations, training and adjustment parameters, data processing and pre-processing	69.5%
6	Reference standard described in detail	Evaluation of the sample selection method, data labeling, description of classification categories, and agreement between researchers	66.1%
7	Results interpreted without knowledge of index test	Professional performance in the data classification process was independent of the automated classification	100.0%
8	Reference standard results interpreted without knowledge of index test	Automated classification performance of the data was independent of human classification	100.0%
9	Uninterpretable/intermediate results reported	Description of data that was misclassified in the output of AI	64.4%
10	Withdrawals explained	Description of data inclusion and exclusion criteria	69.5%

*Quality score: Percentage of articles that satisfactorily meet the evaluated item

Results

A total of 1024 articles were found, excluding articles that were repeated in all researched locations. After reading the titles and abstracts, 97 articles were selected for reading in full. After reviewing the 97 studies, 60 articles met the inclusion criteria and were included in the review (Fig. 1).

Fig. 1.

Flowchart summarizing the search strategy

Characteristics of the Selected Studies

Fifty percent of the studies were carried out in the Asian continent (50.8%), with China being the country that had the most productions with this theme (25.4%), followed by Singapore (8.4%) and Korea (6.7%). The other studies are distributed among the American (13.5%) and European (13.5%) continents, and some studies were intercontinental multicenter (13.5%).

The publication dates of the 60 selected articles varied between the years 2003 and 2021. At the end of 2018, we identified ten studies (16.9%), in 2003 [51], in 2016 [25], in 2017 [24, 52, 53], and in 2018 [23, 26, 54–56]. The predominance of publications occurred from the year 2019, where we identified 14 studies (23.7%) [6–8, 18, 22, 28, 34, 35, 57–62]; in 2020 there were 17 publications (27.11%) [3–5, 9, 10, 16, 17, 26, 31, 32, 36, 48, 63–67]; and in 2021, there were 18 studies (32.2%) [1, 10–15, 33, 68–78].

Resources involving AI can be applied in several areas of health. In this study, specialties with potential similarities were considered, which use clinical images as a complementary registration document in the routine of the services. The articles were in the field of ophthalmology (66.6%) (Table Supplementary 2) [6–18, 22–25, 31–33, 51, 53, 54, 56, 58–61, 63, 64, 68–78], dermatology (15%) (Table Supplementary 3) [4, 5, 26–28, 34–36, 71], and oral medicine (18.3%) (Table Supplementary 4) [1, 3, 48, 52, 55, 57, 62, 65–67, 79]. The use of instruments that make it possible to obtain images was admitted, such as the ophthalmoscope in ophthalmology and the dermatoscope in dermatology.

To enable the analysis of advances obtained with the use of AI in the automated classification of lesions using clinical images in ophthalmology, dermatology, and oral medicine, it was opted for leaving aside the established criteria. In the specific case of oral medicine, studies were included when the automated classification of clinical images was tested, although without the comparison between the performance of the machine and the human comparator, which resulted in the inclusion of 8 additional studies (15%) [48, 52, 55, 57, 62, 65, 67, 79].

Approximately 40 types of architectures, used alone or in combination, were found in the studies. Most frequent were (a) InceptionV3 [6, 25, 34, 54, 56, 58, 61, 68, 69, 73], U-Net20 [11, 17, 23, 59, 79], and ResNet-50 [22, 78] in ophthalmology; (b) DenseNet alone [17, 63, 64, 72] or in combination with InceptionV3 [34] in dermatology; and (c) U-Net4 ResNet-50 [57, 79] in oral medicine. Some technologies, which undergo updates to meet new demands, were Inception V1 [11, 23, 59], ResNetV2 initiation [34], InceptionV3 [6, 34, 54, 56, 58, 61, 68, 69, 73], and Inception-v44 [5, 62, 68].

A great diversity can be seen in the composition of the research teams that carried out the collection and labeling of the data and the architecture training. The composition of the research teams ranged from one [24, 51], two [17, 31, 32, 61, 71], three [7, 11, 16, 35, 53, 60, 64, 68, 77], four [14, 15], six [22, 63, 75], and eight researchers [10, 23]. Collections and labels were performed by nurses, technicians [8], or students [74] but supervised by ophthalmologists. The performance of AI was compared with specialists [8, 74], students [12, 69, 73], trained graders [13], and ophthalmologists [12, 69, 73]. The experienced ophthalmologists did the discrepancy assessment [12, 13, 71]. Large teams ranging from 20 to 67 professionals with varying degrees have been described [3–6, 10, 25, 26, 34, 36, 54, 56]. Some studies did not describe the research team [59, 70, 76].

A frequent technique in the studies was the pre-processing of the database images before development and testing the AI architectures. The description of pre-processing in 60% of ophthalmology studies, 55% of dermatology studies, and 90.9% of oral medicine studies was observed (Table 2).

Table 2.

Artificial intelligence model performance

Specialty		Ophthalmology (n = 40)	Dermatology (n = 9)	Oral medicine (n = 11)
Performance	Superior to experts	1	1	1
	Equivalent	30	5	1
	Equivalent and superior to non-expert	2	2	1
	Equivalent and inferior to expert	2	1	0
	Inferior to experts	5	0	0
	Do not compare with humans	0	0	8
Pre-processing	Yes	24	5	10
	No report	16	4	1
Image criteria	Yes	36	6	10
	No report	4	3	1
Public Bank	Yes	15	4	0
	No	25	5	11

Most studies (86.6%) describe the existence of protocols for collecting clinical images, and regarding the use of images obtained from public databases, 37.5% of the studies of ophthalmology and 44.4% of the dermatology studies used these sources to build their research database. Among the oral medicine studies, 100% used their own databases for the development of the studies (Table 2).

The performance of the model in the automated classification of images was evaluated, compared to human classification. Categories were created to report the findings, as can be seen in Table 2. It was observed that in 5% of studies [34, 61, 66], the AI model surpassed the experts, 60% of studies showed AI performance equivalent to expert performance, and in 8.3% of cases [3–5, 58, 78], AI overcame only non-specialized professionals. In 13.3% of cases, AI did not surpass the human experts. In another 13.3% of the studies, there was no comparison with humans; all studies were of oral medicine [48, 52, 55, 57, 62, 65, 67, 79].

Discussion

The present study carried out a systematic review of the literature on the effectiveness of AI techniques based on DL and artificial neural networks for automated classification of clinical images, when compared to human performance. Despite the evident potential to help health professionals to identify different diseases, the preliminary results indicate that more studies are needed to resolve some critical points and make present sufficient security for the clinical application of these technologies in clinical practice.

The results of the review show that only 5% of the studies demonstrate an AI model performance that outperforms human comparators and in 60% of cases the performance of the model that equals human. Among the studies with performance equivalent to the human comparator, 24 of the 36 studies (66.6%) reported protocols for image collection, as well as all those that surpassed the human comparator. Despite this encouraging result, it is necessary to observe the research scenario that must reproduce the real-life scenario to be generalized.

Image quality is highlighted as a critical point for automatic classification technologies to get high precision [23, 27, 31, 36, 60, 63, 72, 76, 78]. Evidence point the need to maintain the image quality control mechanisms in the development of technologies, for the applicability of the model in clinical practice [23]. Deshmukh et al. [70] demonstrated a semiautomatic model that allows manual corrections when the technology performs imperfect segmentations recognizing the importance of human participation in checking the automated process.

Some AI models are designed to work online with the proposal of a preliminary classification in real time. If the image obtained does not meet the minimum criteria established for the good performance of the technology, a new image is requested [12, 15, 71]. However, the performance of AI models may require greater computational power and server for hosting deployment and operation data; this was a reported limitation in a country with few technological resources [8]. This is an important finding, since many studies idealize the benefit of this resource for remote regions with limited structures and lack of professionals [3–7, 10, 12, 33, 66, 69, 71, 74, 77]. Performance validation of real-time AI software still requires studies and testing [56].

Despite good results in image classification, generalizability to clinical practice may be compromised if the training database does not include the heterogeneity of lesions [3, 28]. The ophthalmology has a standardized protocol for obtaining clinical images. Although this presents a restricted anatomical field, there are many structural variations and visually similar pathologies that can manifest in the same region of interest [16, 32, 59]. Despite this, the performance of the models used in the field of ophthalmology was predominantly equivalent to human comparators, only 7 of the 40 (17.5%) studies had inferior performance but still with high sensitivity and specificities [13, 54, 60, 64].

A superficial analysis could consider that clinical dermatology images configure easy access and present visual patterns easily recognized by AI architectures, as they are often well defined and homogeneous. However, in addition to the various skin pathologies [4, 5], it was noticed that the technology was greatly influenced by aspects such as background variations, lighting, distance, and image capture angle [7, 27, 35]. They have also been reported due to the presence of skin appendages, skin tones in different ethnicities, and even the anatomical region involved [28, 34, 71].

When considering images of visually similar lesions, Zhen et al. [34] observed confounding factors, demonstrating that AI has a limit for classifying similar lesions. To overcome this barrier, the model by Liu et al. [5] established three outputs as hypotheses for diagnosing the lesion analyzed, greatly improving the accuracy of the AI. According to the authors, this method can alert professionals to differential diagnoses that they may not have considered, which can be significant in clinical decision-making [5]. Stereophotogrammetry is a tool that has been proposed for the accurate assessment of skin lesions that project volume, such as those induced by HPV. Future studies should explore the examination of 3D photography to assess lesion size and response to treatment [80].

Multimedia reports and integration with health records remain evolving, such as Digital Imaging and Communications in Medicine (DICOM). Standards can be adopted to facilitate work steps such as creating, viewing, and exchanging reports, with markings and annotations that can contribute to communication between professionals and to monitoring the evolution of identified lesions [19].

A process widely accepted to homogenize data is pre-processing. This step makes it possible for standardization of input data in the AI model, the manipulation, or removal of artifacts [33]; edge clipping [9, 16, 25, 57, 63, 69, 72, 74, 75], color manipulation, lighting, contrast, and sharpness [3, 16, 26, 31, 33, 69, 70, 73, 74, 78]; data augmentation through techniques such as rotational scaling and spatial variations [3, 5, 16, 31, 33, 57, 59, 61, 63, 64, 70, 72, 74, 75, 78]; transfer of learning [48, 57, 79]; standardize the resolution of images [3, 6, 9, 26, 33, 57, 59, 69, 70, 73]; resizing [3, 5, 6, 9, 16, 31, 33, 54, 69, 72, 74, 75]; and also metadata encoding [5].

Regarding the oral medicine, 90% of the studies describe standardization methods of data collection and pre-processing techniques. To standardize the collection of images, the use of a camera with a specific configuration, protocols defining the degrees of inclination in capturing images [3, 66], and drying of the mucosa to reduce the reflex [65] were identified. Among the pre-processing techniques cited are cropping the area of interest [57, 62, 65–67, 79] to avoid confounding factors such as teeth and dental instruments [79], resizing [55, 65, 66, 79], contrast adjustment, reflection removal [62, 65], and resolution standardization [55]. Considering the application of these technologies in real-life practice, it seems feasible to standardize the pre-processing step. On the other hand, optimal image capture setup is a critical issue and is generally more difficult to implement in real-life applications.

In oral medicine, there is a variety of designs. The studies usually have a relatively high accuracy rates in automated classification, but some aspects should be highlighted. The study by Fu et al. was the only one that had a larger development sample, with 5775 images of benign and malignant lesions in different stages and in various anatomical sites of the oral cavity as well as controls without lesion, obtained from 11 Chinese hospitals. Classification into cancer, non-cancer, and healthy mucosa reached an AUC that ranged from 0.93 to 0.99 [3]. Tanriver et al. used a mixed database, in which part of the data had histopathological confirmation and the rest was obtained from the web. Its sample consisted of 652 images of benign, potentially malignant, and malignant lesions with variations in image quality and anatomical points. Based on the selection of an area of interest, it obtained an average accuracy of 0.87 [79].

In other previous study, the database contained 2155 images with and without injury, part received from specialists and part obtained from the Web, with no restriction regarding the standard and quality of the images, demonstrating the difficulty of obtaining a broad database. In a binary analysis, they achieved a precision of 84.77 for identifying the existence of a lesion, 67.15 for the need for referral to a specialist, and 52.13 for classifying it as benign, malignant, or potentially malignant. There was a significant reduction in the multiclass analysis, precision of 46.61 for identifying the existence of a lesion, 32.97 for the need for referral to a specialist, and 17.71 for classifying it as benign, malignant, or potentially malignant, demonstrating that a critical point is the heterogeneity of the lesions that appear in the oral cavity [48].

In general, studies demonstrated high precision, but with specific pathological conditions, not very comprehensive and reduced databases, with part of the data obtained on the internet. Shamim et al. used a sample of 200 images. Classifying conditions that manifest themselves in the tongue, it obtained an accuracy of 0.93 [57]; Jurczyszyn et al. used a sample of 63 patients, divided into leukoplakia, lichen planus, and healthy. Clinical images were obtained using an image capture and histological confirmation protocol. It obtained the best results in the comparison between lesion and normal mucosa, however, with little expressive result in the differentiation between leukoplakia and lichen planus (sensitivity of 57%, 38%, and 94% and specificity of 74%, 81%, and 88% for leukoplakia, lichen planus, and healthy mucosa, respectively) [62].

Due to the heterogeneity of oral lesions and the clinical similarity between some pathological entities [3, 48, 62], the diagnostic process in oral medicine goes far beyond the analysis of the characteristics of an image. This process requires the search for strategies that allow the combination of information that can be recognized through imaging and inclusion of additional information obtained through clinical examination. No study about oral diseases describes the use of public databases, which indicates the inexistence of this type of database and justifies the decision to Welikala et al. [48] in creating a database with several oral pathological entities, together with the respective clinical information [48]. Due to lack of public databases, or even lack of collection, some oral medicine studies used web mined images for the development of AI models, obtaining a small and unrepresentative volume of data [48, 52, 55, 57, 79].

The understanding of the automated classification process is an important step, especially in the health area. Although the connections occur in the intermediate layers of the DL models and are unknown [81], the error analysis can give clues to the data that generate confusion. This analysis can improve the understanding of the parameters that the models use to make their decisions [81–83]. Of the studies analyzed, 36.6% identified errors [3, 6, 8, 9, 13, 15, 17, 18, 22, 23, 54, 56, 57, 63, 66, 69, 70, 73, 74, 76–79], but only 23.3% perform some analysis of these errors [6, 9, 13, 17, 18, 22, 56, 63, 69, 70, 73, 74, 78, 79].

Although studies consider the use of AI as an important and promising tool to aid decision-making in clinical practice [48, 57, 79], it does not seem realistic to consider the replacement of specialist professionals. So far, studies show that in the future, this technology can help healthcare networks [57, 65, 79], enabling the screening process in areas of difficult access and optimizing time and the work process of different professionals.

Our review has limitations that should be recognized. Although most of the studies were of good quality when evaluated individually in this study (60%), the heterogeneity of the methodologies used does not allow their use in a meta-analysis. Owing to the heterogeneity of the methodologies used in the studies included in this review, in a rapidly changing field, it is also difficult to generalize the results. Taking the lack of robust evidence into account, further studies are necessary, particularly in different countries, to strengthen the evidence about the application of these features in real life.

Conclusion

Recent studies have generated remarkable advances. The knowledge produced so far has determined the factors that need to be explored to improve the performance of the architectures. In the coming years, the great challenges will be to achieve a satisfactory performance of AI in the identification of lesions at an early stage and the interpretability of the automated classification process for the understanding of the parameters that the models use to make their decisions. Above all, it is essential to be aware that, at least until this moment, one should not create the expectation of creating a tool that will replace human evaluation but may be an adjunct for the clinical setting, particularly for health professionals with little training for recognition of a specific disease. The pre-processing seems to be a predictable technical step, which allows the reduction of the variability of the data presented to the artificial intelligence, enhancing the performance of automated classification in the real-life multicentric scenario.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contribution

RFTG, RMF, LFS, and VCC conceived, designed, guided, and coordinated the study and the writing. RFTG and EFH identified publication records from MEDLINE and screened the titles, abstracts, and full text of the articles. Rita identified additional articles that were not retrieved in MEDLINE. RFTG and VCC prepared the figures and tables. All the authors contributed to the writing of the article. The Abstract, Introduction, Materials and Methods, Discussion, and Conclusion sections were written jointly by RFTG and VCC. RFTG and VCC performed thorough editing of the article. All the authors revised and approved the final article.

Funding

No funding was obtained for this study.

Data Availability

Not applicable.

Declarations

Ethics Approval

Ethical approval for this type of study is not required by our institute.

Conflict of Interest

The authors declare no competing interests.