Abstract
Background: After the outbreak of the corona virus disease-19 (COVID-19) pandemic, teledermatology was implemented in the Hungarian public healthcare system for the first time. Our objective was to assess aggregated diagnostic agreements and to determine the effectiveness of an asynchronous teledermatology system for skin cancer screening. Methods: This retrospective single-center study included cases submitted for teledermatology consultation during the first wave of the COVID-19 pandemic. Follow-up of the patients was performed to collect the results of any subsequent personal examination. Results: 749 patients with 779 lesions were involved. 15 malignant melanomas (9.9%), 78 basal cell carcinomas (51.3%), 21 squamous cell carcinomas (13.8%), 7 other malignancies (4.6%) and 31 actinic keratoses (20.4%) were confirmed. 87 malignancies were diagnosed in the high-urgency group (42.2%), 49 malignancies in the moderate-urgency group (21.6%) and 16 malignancies in the low-urgency group (4.6%) (p < 0.0001). Agreement of malignancies was substantial for primary (86.3%; κ = 0.647) and aggregated diagnoses (85.3%; κ = 0.644). Agreement of total lesions was also substantial for primary (81.2%; κ = 0.769) and aggregated diagnoses (87.9%; κ = 0.754). Conclusions: Our findings showed that asynchronous teledermatology using a mobile phone application served as an accurate skin cancer screening system during the first wave of the COVID-19 pandemic.
Keywords: teledermatology, telemedicine, telehealth, COVID-19, skin cancer, melanoma, skin cancer screening, store-and-forward, diagnostic concordance, accuracy
1. Introduction
After the outbreak of the corona virus disease-19 (COVID-19) pandemic, restrictions were introduced all around the world in early 2020 [1,2]. Health care systems were under tremendous pressure and experienced lack of resources with the burden of the treatment of patients with COVID-19 [3]. Hospitals were also among the riskiest places where the pandemic could gain momentum [4]. For these reasons, outpatient care was restricted, most consultations were postponed for non-life-threatening conditions to prevent the spread of COVID-19 [5]. Emergency conditions were the only exceptions [6]. The use of telemedicine has come forward as an appropriate method to deal with otherwise postponed and non-urgent cases [7,8]. A patient visit where the parties only meet in an online platform, telemedicine, involves the use of various communication technologies to transmit medical data [9,10]. During the first wave of the COVID-19 pandemic, the use of telemedicine spread across different medical disciplines at an unprecedented level, including dermatology as well [11,12,13,14,15].
Different teledermatology services have been used routinely since the mid 2000’s in the public health care system of European countries, such as the United Kingdom, the Netherlands and Spain [16,17]. Despite the fact, high percentage of dermatologists used teledermatology the first time after the outbreak of the COVID-19 pandemic [18,19,20]. Teledermatology is currently used in three main ways [9]. The most widespread modality is asynchronous consultation which uses store-and-forward technologies. It relies on digital photographs of the skin lesions and history provided by the patients. Then data are transmitted to a dermatologist who performs the consultation subsequently [9,21]. The second form, synchronous consultation, which is also commonly used, allows real-time interaction between patients and dermatologists with the use of web cameras or mobile phone cameras [22,23]. The third, hybrid method, is a combination of the previous two techniques [24].
In addition to its widespread use during the pandemic, teledermatology could serve as an effective tool for the early diagnosis of skin cancer [25]. The negative impacts of COVID-19 pandemic have been already reported in the management of skin cancers [26,27]. It can only be estimated that during the first wave of the pandemic how many patients were affected by a delayed diagnosis and treatment with skin cancers [28,29,30]. In skin cancer screening, new methods of early detection has appeared during the past years due to the technical improvements of mobile phone cameras [31]. Studies were published prior to the COVID-19 pandemic that assessed the accuracy of store-and-forward type teledermatology based on photographs taken by mobile phones [32]. One additional public health aspect of store-and-forward teledermatology is that it can serve as a fast and cost-effective triage tool to reduce wait time and improve access to health care [33,34].
The aims of the present study were to evaluate the effectiveness of an asynchronous teledermatology system for skin cancer screening care and to assess aggregated diagnostic agreement.
2. Materials and Methods
2.1. Patient Data
The present study was performed at the Department of Dermatology, Venereology and Dermatooncology, Semmelweis University (Budapest, Hungary). This retrospective and single-center study included patients who submitted their cases for teledermatology consultation during the first wave of the COVID-19 pandemic, between 25 March 2020 and 13 July 2020. Asynchronous teledermatology consultations provided direct store-and-forward type visits between patients and dermatologists. To standardize the procedure, the submission of the cases was carried out by the use of a mobile phone application developed by MedInnoScan Research and Development Ltd. (Budapest, Hungary), which was made available nationwide for free of charge. The application required patients to fill in a questionnaire and send five to 10 photos of the skin lesion(s). The questionnaire included specific questions about the patients’ general health and the current complaints related to the skin lesion(s). At least one of the images had to be a photograph from a distance, to provide information about the surrounding skin adjacent to the lesion. The other images had to be close-ups, from different angles. Photos taken by patients in JPEG format could be uploaded if a resolution of 8 megapixels is achieved. The application went through continuous testing before it was released to the patients.
At the same time with the development of the application, an online interface for personal computers was created. This allowed dermatologist to view photographs and the questionnaire submitted by patients on personal computers. After uploading to the online interface, a specialist first carried out a preliminary assessment to determine if the cases required immediate teledermatology consultation. If the case was not urgent, the consultation was completed within a maximum of three business days. Then the case was treated by a specialist or a specialist registrar. Registrars were only allowed to care for patients under the supervision of a specialist. Patients were registered in the local hospital information system (HIS) (e-Medsolution, T-Systems Hungary Ltd., Budapest, Hungary) and the medical documentation was also completed there. The document included one or more possible differential diagnoses of the lesion(s) and the urgency of the case. If the dermatologists considered that the quality of the images sent by the patients were inadequate and could make it difficult to establish the correct diagnosis, it was noted as well. The medical documentation was sent to the patients by e-mail and it was also uploaded into the National eHealth Infrastructure system of Hungary (EESZT, see http://www.eeszt.gov.hu (accessed on 16 January 2022)). The EESZT provides access to data from all doctor’s health-related visits in public health institutions nationwide.
For the current study, medical records of teledermatology consultations were reviewed to obtain information on patient demographics, diagnosis, and urgency of cases who were sent for personal dermoscopy examination. Follow-up of patients was performed between 1 March and 30 April 2021, to collect the results of face-to-face (FTF) or potential histopathological examinations. Data were collected from the HIS and the EESZT. If relevant information was not found, patients were contacted by phone.
2.2. Diagnostic and Triage Groups
When disease categories were established, all types of malignant melanoma (MM) were placed into one diagnostic group. Similarly, all forms of basal cell carcinomas (BCC) and squamous cell carcinomas (SCC) were placed into one group, respectively. In our study, actinic keratoses (AK) were listed as a malignant lesion as well. The category of other malignancies included skin cancers that could not be accurately identified as MM, BCC, SCC, or AK. Considering non-malignant lesions, dysplastic naevi are represented as a separate group, while all other naevi were grouped together as “naevi”. Seborrheic keratoses (SK), haemangiomas and warts were defined as separate groups. All additional diagnoses with lower numbers of cases were grouped into the “other lesions” group. Table 1 summarizes the diagnostic groups involved in the study.
Table 1.
List of diagnostic groups.
| Diagnostic Groups |
|---|
| Malignant melanoma |
| Squamous cell carcinoma |
| Basal cell carcinoma |
| Other malignancies |
| Actinic keratosis |
| Dysplastic naevi |
| Naevi |
| Seborrheic keratosis |
| Haemnagioma |
| Warts |
| Other lesions |
All cases were classified into three different triage groups by the immediacy of the findings during teledermatology consultations: high-urgency group (immediate FTF examination required), moderate-urgency group (FTF examination in short time) and low-urgency group (FTF examination can be postponed after pandemic) (Figure 1). If more than one lesion was referred to teledermatology consultation by the same patient, these cases received only one single triage status. In our study, this triage status was assigned to both lesions, equally.
Figure 1.
Definition of the triage groups. Abbreviations: FTF, face-to-face.
2.3. Outcome Measures
The primary outcome measure was to determine the aggregated diagnostic agreement of the various diagnostic groups and the overall system, and to determine the efficacy of store-and-forward teledermatology as a skin cancer screening tool. Aggregated diagnostic agreement is defined as agreement of the primary diagnosis or any of the differential diagnoses considered during teledermatology consultation with the results of FTF dermoscopy examination or histopathology [35]. Distribution of true positive and false negative diagnoses were also evaluated among different triage groups, based on aggregated diagnosis of the lesions. Overall primary diagnostic agreement was also measured, which is the agreement between the primary diagnosis during teledermatology consultation and the results of FTF or histology examination [36].
2.4. Statistical Analyses
Descriptive statistics were reported for patient demographics and lesion localizations. Pearson’s chi-square test (two-tailed) was used for categorical variables. Contingency tables were used to determine diagnostic parameters, such as sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the different diagnostic groups. Cohen’s kappa coefficient (κ) was assessed for each diagnostic group separately to determine the concordance between the diagnosis established during teledermatology consultation and the reference standard. Based on the guideline, κ ≤ 0.2 indicates slight agreement, κ of 0.21–0.40 indicates fair agreement, κ of 0.41–0.60 indicates moderate agreement, κ of 0.61–0.8 indicates substantial agreement, while κ of 0.81–1.00 indicates almost perfect agreement [37]. Overall diagnostic accuracy and concordance were calculated separately for all malignant and non-malignant lesions, as well as for all lesions included in the study. Confidence intervals (95% CI) were calculated whenever appropriate. Statistical analyses were performed using Statistica v13.5.0.17 software (TIBCO Software Inc., Palo Alto, CA, USA). κ values were assessed using GraphPad QuickCalcs calculator (GraphPad Software Inc., San Diego, CA, USA).
Two reference standards were defined in this study. For lesions from which histology was performed after the FTF examination, histology was the reference standard. Where no histology was performed, the reference standard meant the result of the FTF examination.
2.5. Inclusion Criteria
All patients whose photographs were deemed of sufficient quality by dermatologists during teledermatology consultations and attended personal dermoscopy examination were included in this study. An additional criterion was the availability of results from the FTF examination or histology if the latter was performed.
2.6. Exclusion Criteria
Patients were excluded if the dermatologists could not determine the diagnosis during teledermatology consultation due to lack of proper photographs or absence of medical history. Patients were also excluded from the study if no follow-up information could be obtained.
3. Results
3.1. Patient Data
1447 patients with 1495 lesions were sent for dermoscopy examination via store-and-forward teledermatology consultation. 124 patients with 124 lesions were excluded before follow-up because of the low quality of the photographs. In 194 cases, the results of the personal examinations were found in the HIS. In 200 cases, the data were found in the EESZT. To obtain additional information, 929 patients were contacted by phone. After the follow-up, 83 patients with 83 lesions were excluded due to lack of information. 491 patients with 509 lesions did not attend FTF examination and consequently they were also excluded.
A total of 749 patients with 779 lesions were included in our study as they underwent FTF examination. Patient characteristics are shown in Table 2. The mean age of the included patients was 43.54 ± 21.03 years. The median number of lesions per patient was one (interquartile range: 1–2). 45 patients (6%) had skin cancer in the personal history, 18 patients (2.4%) had in the family history, while 2 patients (0.3%) had in both. The remaining 684 patients (91.3%) had no skin cancer in their medical history. During teledermatology consultation sessions, 639 (82%), 132 (17%), and 8 lesions (1%) received a single, two and three diagnoses, respectively. Teledermatology consultations were carried out by 29 dermatologist specialists and specialist registrars. The average number of completed cases per dermatologist was 25.8 ± 17.8. Specialist registrars, under the supervision of a specialist, cared for 565 patients with 586 lesions (75.2%). 184 patients with 193 lesions (24.8%) were managed exclusively by a specialist.
Table 2.
Characteristics of the included patients.
| Variables | No. |
|---|---|
| Age composition | |
| 0–19 | 87 (11.6%) |
| 20–39 | 225 (30.0%) |
| 40–59 | 245 (32.7%) |
| 60–79 | 164 (21.9%) |
| 80≤ | 28 (3.7%) |
| Sex | |
| Female | 474 (63.3%) |
| Male | 275 (36.7%) |
| Ethnicity | |
| Caucasian | 744 (99.3%) |
| Others | 5 (0.7%) |
| Lesion location | |
| Head/neck | 193 (24.8%) |
| Hand/arm | 117 (15.0%) |
| Trunk | 350 (44.9%) |
| Leg/foot | 103 (13.2%) |
| Buttock/groin | 16 (2.1%) |
Abbreviations: No., number.
3.2. Triage Groups
Figure 2 shows the distribution of triage groups among true positive and false negative diagnoses. 206 lesions (26.5%) were triaged as high-urgency, 227 lesions (29.1%) as moderate-urgency, and 346 lesions (44.4%) as low-urgency during teledermatology consultations, respectively. Two separate lesions were referred to teledermatology consultation by the same patient in 30 cases. The same triage status was assigned equally to both lesions.
Figure 2.
Distribution of different triage groups among true positive (Panel A) and false negative diagnoses (Panel B), considering aggregated diagnoses of the lesions during teledermatology consultations.
Significant differences were found between different triage groups in the number of true positive diagnoses (p < 0.0001). 156 lesions (75.7%) were diagnosed correctly in the high-urgency group, 195 (85.9%) lesions in the moderate-urgency group and 334 lesions (96.5%) in the low-urgency group, considering aggregated diagnoses. The reference standard was the result of the FTF examination in 509 cases and the histological examination in 270 cases. The distribution of confirmed malignancies showed also significant differences between triage groups. After the follow-up, 87 cases of malignancies were confirmed in the high-urgency group (42.2%), 49 malignancies in the moderate-urgency group (21.6%) and 16 malignancies in the low-urgency group (4.6%), respectively (p < 0.0001) (Figure 2).
3.3. Overall Primary and Aggregated Diagnostic Agreement
Table 3 reveals the overall primary and aggregated diagnostic agreement of malignant lesions, non-malignant lesions and all lesions. In terms of all malignant lesions, diagnostic concordance indicated substantial agreement. In contrast, higher concordance values were calculated for non-malignant lesions as they showed almost perfect agreement considering primary diagnosis, while aggregated diagnostic concordance indicated substantial agreement. The overall concordance between all teledermatology consultations and the reference standard showed substantial agreement. In total, significant difference was found between overall aggregated and primary diagnostic accuracy (p < 0.0001) (Table 3). The primary diagnosis of 633 lesions during teledermatology consultations matched with the reference standard, while 146 lesions were misdiagnosed. When all differential diagnoses were included, 685 lesions matched with the reference standard and 94 lesions did not.
Table 3.
Overall diagnostic agreement.
| PD/AD | Accuracy (95% CI) |
Cohen’s Kappa (95% CI) |
|---|---|---|
| Malignant lesions | ||
| PD | 86.3% (84.1–88.7%) |
0.647 (0.574–0.720) |
| AD | 85.3% (82.9–87.9%) |
0.644 (0.572–0.716) |
| Non-malignant lesions | ||
| PD | 81.3% (78.6–84.0%) |
0.811 (0.790–0.830) |
| AD | 86.5% (84.1–88.9%) |
0.790 (0.769–0.810) |
| Total lesions | ||
| PD | 81.2% (78.4–83.8%) |
0.769 (0.747–0.792) |
| AD | 87.9% (85.5–90.0%) |
0.754 (0.722–0.776) |
Abbreviations: PD: primary diagnostic, AD: aggregated diagnostic, CI: Confidence Interval.
3.4. Diagnostic Groups
The possibility of a malignancy was considered in 198 lesions based on primary diagnosis, while it increased to 228 lesions, according to aggregated diagnosis. In case of 12 lesions, the chance of two different malignancies was raised by teledermatologists. Altogether, 152 patients (male/female: 78/74; mean age: 62.26 ± 16.13 years) were diagnosed with malignancy after personal examination. Histology examination confirmed the diagnosis of MM in 15 cases, BCC in 78 cases, SCC in 21 cases, other malignancies in seven cases and AK in three patients. In case of the remaining 28 AK diagnoses, the reference standard was the result of the FTF examination. After histological examination, the following malignancies were confirmed in the other malignancies group: metastasis of Merkel cell carcinoma (1 lesion), metastasis of adenocarcinoma (1 lesion), dermatofibrosarcoma protuberans (1 lesion), invasive mammary carcinoma (1 lesion), primary cutaneous follicle center lymphoma (3 lesions). In case of malignant lesions, sensitivity values ranged from 64.5% to 89.7% (κ = 0.410–0.770) according to primary diagnoses. Sensitivity values improved to 71.4–93.3% (κ = 0.485–0.714), considering aggregated diagnoses (Table 4). Among all cases of confirmed malignancies, 24 patients (15.8%) had a history of previous skin cancer, while 6 patients (3.9%) had at least one close relative diagnosed with skin cancer. 113 patients (80.3%) did not mention any related information during teledermatology consultation.
Table 4.
Diagnostic parameters and concordance of malignant diagnostic groups.
| PD/AD | No. of Diagnoses during TDC | TP | FN | Cohen’s Kappa (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) |
NPV (95% CI) |
|---|---|---|---|---|---|---|---|---|
| Malignant melanoma | ||||||||
| PD | 32 | 10 | 5 | 0.410 (0.231–0.589) |
66.7% (41.7–84.8%) |
97.1% (95.7–98.1%) |
31.3% (18.0–48.6%) |
99.3% (98.44–99.7%) |
| AD | 42 | 14 | 1 | 0.476 (0.317–0.636) |
93.3% (70.2–99.7%) |
96.3% (94.8–97.5%) |
33.3% (21.0–48.5%) |
99.9% (99.2–100.0%) |
| Squamous cell carcinoma | ||||||||
| PD | 36 | 13 | 8 | 0.437 (0.273–0.600) |
61.9% (40.9–79.3%) |
97.0% (95.5–98.0%) |
36.1% (22.5–52.4%) |
98.9% (97.9–99.5%) |
| AD | 45 | 19 | 2 | 0.560 (0.415–0.704) |
90.5% (71.1–98.3%) |
96.6% (95.0–97.7%) |
42.2% (29.0–56.7%) |
99.7% (99.0–100.0%) |
| Basal cell carcinoma | ||||||||
| PD | 98 | 70 | 8 | 0.770 (0.698–0.842) |
89.7% (81.1–94.7%) |
96.0% (94.3–97.2%) |
71.4% (61.8–79.4%) |
98.8% (97.7–99.4%) |
| AD | 112 | 71 | 7 | 0.714 (0.638–0.789) |
91.00% (82.6–95.6%) |
94.2% (92.2–95.7%) |
63.4% (54.2–71.7%) |
99.00% (97.9–99.5%) |
| Other malignancies | ||||||||
| PD | 8 | 5 | 2 | 0.663 (0.386–0.941) |
71.4% (35.9–94.9%) |
99.6% (98.9–99.9%) |
62.5% (30.6–86.3%) |
99.7% (99.1–99.9%) |
| AD | 8 | 5 | 2 | 0.663 (0.386–0.941) |
71.4% (35.9–94.9%) |
99.6% (98.9–99.9%) |
62.5% (30.6–86.3%) |
99.7% (99.1–100.0%) |
| Actinic keratosis | ||||||||
| PD | 25 | 20 | 11 | 0.704 (0.566–0.842) |
64.5% (47.0–78.9%) |
99.3% (98.4–99.7%) |
80.0% (60.9–91.1%) |
98.5% (97.4–99.2%) |
| AD | 33 | 24 | 7 | 0.739 (0.617–0.862) |
77.4% (60.2–88.6%) |
98.8% (97.7–99.4%) |
72.7% (55.8–84.9%) |
99.1% (98.1–99.5%) |
Abbreviations: PD: primary diagnostic, AD: aggregated diagnostic, No.: number, TDC: teledermatology consultations, TP: true positive, FN: false negative, CI: confidence interval, PPV: positive predictive value, NPV: negative predictive value.
Considering non-malignant lesions, histology examination was the gold standard in 146 cases, while FTF examination confirmed the diagnosis in 481 cases. In our study, the naevi diagnostic group included the largest number of lesions during teledermatology consultations. Concordance of naevi indicated almost perfect agreement, while the lowest agreement was assessed for dysplastic naevi, fair and moderate agreements were assessed. In case of non-malignant pigmented lesions, sensitivity values ranged from 80% to 91.2% (Table 5).
Table 5.
Diagnostic parameters and concordance of dysplastic naevi and naevi groups.
| PD/AD | No. of Diagnoses during TDC | TP | FN | Cohen’s Kappa (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) |
PPV (95% CI) |
NPV (95% CI) |
|---|---|---|---|---|---|---|---|---|
| Dysplastic naevi | ||||||||
| PD | 38 | 12 | 3 | 0.437 (0.270–0.605) |
80.0% (54.8–93.0%) |
96.6% (95.1–97.7%) |
31.6% (19.1–47.5%) |
99.6% (98.8–99.9%) |
| AD | 46 | 12 | 3 | 0.375 (0.220–0.530) |
80.0% (54.8–93.0%) |
95.50% (93.9–96.8%) |
26.10% (15.6–40.3%) |
99.6% (98.8–99.9%) |
| Naevi | ||||||||
| PD | 265 | 248 | 37 | 0.848 (0.809–0.887) |
87.0% (82.6–90.4%) |
96.6% (94.6–97.8%) |
93.6% (90.0–96.0%) |
92.8% (90.2–94.7%) |
| AD | 290 | 260 | 25 | 0.848 (0.810–0.887) |
91.2% (87.4–94.0%) |
93.9% (91.5–95.7%) |
89.7% (85.6–92.7%) |
94.9% (92.6–96.5%) |
Abbreviations: PD: primary diagnostic, AD: aggregated diagnostic, No.: number, TDC: teledermatology consultations, TP: true positive, FN: false negative, CI: confidence interval, PPV: positive predictive value, NPV: negative predictive value.
Concordance of SK (κ = 0.780) and other lesions diagnostic groups (κ = 0.676) indicated substantial agreement. Considering non-malignant diagnostic groups, other lesions had the lowest sensitivity (77.9%) as 15 out of 21 false negative cases were diagnosed as malignant lesions during teledermatology consultations. SK were misdiagnosed as malignant lesions in 14 out of 21 false negative cases. In terms of haemangiomas (κ = 0.961) and warts (κ = 0.943), almost perfect agreement was indicated (Table 6). Distribution of diagnoses of the other lesions diagnostic group is shown in the Supplementary materials (Figure S1).
Table 6.
Diagnostic agreement of SK, haemangiomas, warts and other lesions groups.
| PD/AD | No. of Diagnoses during TDC | TP | FN | Cohen’s Kappa (95% CI) | Sensitivity (95% CI) | Specificity (95% CI) |
PPV (95% CI) |
NPV (95% CI) |
|---|---|---|---|---|---|---|---|---|
| Seborrheic keratosis | ||||||||
| PD | 123 | 112 | 31 | 0.810 (0.754–0.865) |
78.3% (70.9–84.3%) |
98.3% (96.9–99.0%) |
91.10% (84.7–94.9%) |
95.30% (93.4–96.7%) |
| AD | 154 | 122 | 21 | 0.780 (0.723–0.836) |
85.3% (78.6–90.2%) |
95.0% (93.0–96.4%) |
79.2% (72.1–84.9%) |
96.6% (94.9–97.8%) |
| Haemangiomas | ||||||||
| PD | 67 | 65 | 5 | 0.944 (0.903–0.985) |
92.9% (84.3–96.9%) |
99.7% (99.0–100.0%) |
97.0% (89.8–99.5%) |
99.3% (98.4–99.7%) |
| AD | 69 | 67 | 3 | 0.961 (0.926–0.995) |
95.7% (88.1–98.8%) |
99.7% (99.0–100.0%) |
97.1% (90.0–99.5%) |
99.6% (98.8–99.9%) |
| Warts | ||||||||
| PD | 15 | 15 | 4 | 0.880 (0.763–0.996) |
78.9% (56.7–91.5%) |
100% (99.5–100.0%) |
100% (79.6–100.0%) |
99.5% (98.7–99.8%) |
| AD | 17 | 17 | 2 | 0.943 (0.865–1.000) |
89.5% (68.6–98.1%) |
100% (99.5–100.0%) |
100% (81.6–100.0%) |
99.7% (99.1–100.0%) |
| Other lesions | ||||||||
| PD | 72 | 63 | 32 | 0.731 (0.652–0.810) |
66.3% (56.3–75.0%) |
98.7% (97.5–99.3%) |
87.5% (77.9–93.3%) |
95.5% (93.7–96.8%) |
| AD | 111 | 74 | 21 | 0.676 (0.598–0.753) |
77.9% (68.6–85.1%) |
94.6% (92.6–96.1%) |
66.7% (57.5–74.8%) |
96.9% (95.2–97.9%) |
Abbreviations: PD: primary diagnostic, AD: aggregated diagnostic, No.: number, TDC: teledermatology consultations, TP: true positive, FN: false negative, CI: confidence interval, PPV: positive predictive value, NPV: negative predictive value.
4. Discussion
After the outbreak of the COVID-19 pandemic, there was a sudden increase in the demand for teledermatology care [38]. Temiz et al. (2020) were the first to propose that teledermatology could serve as a suitable method for the reduction of FTF consultations during the pandemic [39]. When teledermatology was used prior to the pandemic, photographs were most commonly taken by healthcare professionals using a digital camera or mobile phone [40,41,42,43]. There are just a few publications in the literature of the use of store-and-forward teledermatology consultations where photos were provided by the patients with a mobile phone’s camera and application [44,45,46].
We established a novel store-and-forward teledermatology service at our department to maintain the outpatient dermatology care during COVID-19 pandemic. Upon retrospective analysis of our data, the overall diagnostic concordance indicated substantial agreement (87.9%, κ = 0.754) between teledermatology consultations and the reference standard. In comparison, Moreno-Ramirez et al. (2007) showed an almost perfect agreement (κ = 0.81) in 890 patients [36]. Similarly, Kroemer et al. (2011) reached almost perfect agreement (90%, κ = 0.84) with the combined use of teledermoscopy and macroscopic images in 80 patients with 104 lesions [47]. Lamel et al. (2012) reported substantial agreement between FTF and mobile phone teledermatology evaluation as primary diagnostic concordance was 0.60, while the aggregated diagnostic concordance was 0.62 in 87 patients with 137 lesions [48]. Clarke et al. (2021) used macroscopic digital images taken during FTF examinations which were later sent for teledermatology assessment. They reported moderate agreement (66.6%, κ = 0.60) for primary diagnosis of 308 lesions. Based on these reports in the literature, one can conclude that store-and-forward method where the image acquisition is carried out by the patients display similar concordance to other types of teledermatology.
In previous studies, overall diagnostic accuracy for malignancies varied between 51% and 87.3%, while concordance ranged from moderate to substantial agreement (κ = 0.41–0.63) [49,50,51]. We observed a diagnostic accuracy at the higher end by reaching 85.3%, while the diagnostic concordance was 0.644. Regarding the concordance of different malignancies, we found moderate agreement for MM (κ = 0.475) and SCCs (κ = 0.560), while substantial agreement was seen for BCC (κ = 0.714) and AK (κ = 0.739). This could be related to the fact that higher number of false positive than true positive values were found in cases of MM and SCC (Table 4). Our findings showed similarities with results published by Giavina-Bianchi et al. (2020) [49]. They were using asynchronous technology to screen the population in Brazil before the outbreak of the COVID-19 pandemic and assessed consistent concordances in case of MM (κ = 0.209), SCC (κ = 0.627), BCC (κ = 0.680) and AK (κ = 724).
When all differential diagnoses were considered, high sensitivity was assessed for MM (93.3%), SCC (90.5%) and BCC (91%), similar to data in previous studies [32,47]. From a clinical perspective, false negative diagnosis of MM is a more crucial problem than it is for SCC or BCC [51]. In our set of patients, one nodular melanoma was misdiagnosed as haemangioma during teledermatology consultation. Nevertheless, triaged as a moderate-urgent case, the patient attended FTF examination within a month and the tumor was excised. This case highlights that teledermatologists should be careful with newly developed nodular vascularized or pinkish lesions. It has been shown that diagnostic of non-melanoma skin cancers (NMSC) via teledermatology is significantly hampered if the lesion is at intimate body sites [52]. In addition, in our study, one of the misdiagnosed SCC was localized in the genital region. Other studies have confirmed that the use of dermoscopic images or the combination of macroscopic and dermoscopic images for teledermatology consultations increased the diagnostic accuracy both for pigmented lesions and NMSC [53,54,55]. Contrary to our findings, malignant lesions other than MM, SCC, BCC or AK have been rarely seen in other publications [36,51]. In our study, the diagnosis of other malignancies was correctly established during teledermatology evaluation in five patients, in four of whom the availability of medical history aided the diagnosis. All these malignancies were eventually found to be recurrent tumors. Similarly to our data, Knudsen et al. have emphasized the importance of various telemedicine modalities in the follow-up of oncology patients during the pandemic [56].
Other studies have shown that teledermatology triage services for patients with suspected skin cancer can significantly reduce time to reach the diagnosis [47,57,58]. In our study, vast majority of malignancies have been categorized as to seek immediate help (Figure 2). In the low-urgency group, 14 of 16 cases of skin cancers were correctly diagnosed as AK during teledermatology consultations and the possibility of invasive skin cancer could be ruled out. Dermatologists informed these patients, taking into account the COVID-19 pandemic, that once restrictions have been lifted, they should attend FTF examination. The remaining two lesions, which were both superficial BCC, were diagnosed as SK during teledermatology consultation. Both lesions were removed with adequate safety margin after FTF, and no further treatment were needed.
The highest concordance (κ = 0.848) was calculated for naevi among pigmented lesions, while the sensitivity of this diagnostic group was 91.2%. This means that a significant proportion of non-malignant pigmented lesions have been properly recognized during teledermatology consultations. In case of dysplastic naevi, we advised immediate FTF examination to reduce the chance of misdiagnosis of MM. For this reason, concordance (κ = 0.375) for dysplastic naevi indicated only fair agreement.
In certain cases, the diagnosis of SK turned out to be challenging even during FTF examinations [59]. Furthermore, teledermatologists also consider SK as a potential false positive diagnosis for MM and NMSC [60,61]. In our study, most of the SKs were diagnosed correctly and the concordance (κ = 0.780) indicated substantial agreement that was higher compared to the results (κ = 0.513) of Giavina-Bianchi et al. (2020) [49]. Among all diagnostic groups, haemangiomas showed the highest sensitivity (95.7%) and concordance (κ = 0.961) values. In line with our findings, Betlloch-Mas et al. estimated very high concordance for haemangiomas (κ = 0.924) in the context of paediatric teledermatology care [62]. All the infants with large haemangiomas in our study were referred to FTF examination in a short term to start propranolol treatment and to rule out internal vascular malformations (Figure 2).
Our study has certain limitations. No dermoscopy images could be taken during submission of the cases. The quality of the photographs could be highly variable as they were provided by the patients. Objective inclusion criteria were not determined for the quality of the photographs and it was decided by the dermatologist if they were of sufficient quality. Further diagnostic skin imaging modalities such as high-frequency ultrasonography, optical coherence tomography and reflectance confocal microscopy were not applied for the diagnosis of skin lesions in the present study. In comparison with others, the present study was a real crisis situation setting, so it could not contain management agreement and inter-observer concordance [48,51,63]. Important outcomes such as satisfaction and costs were not addressed in this study. Some studies have excluded patients under the age of 18, while we did not. This could result in higher diagnostic parameters in the diagnosis of skin cancers and other non-malignant lesions [64].
5. Conclusions
To the best of our knowledge, our study was the first to investigate the efficacy of asynchronous teledermatology for the detection of skin cancer in the Central European region. Even though previous plans existed, the outbreak of COVID-19 accelerated our work and allowed real life implementation shortly after the lockdown that also restricted access to doctor’s visits in Hungary. Our findings showed that asynchronous teledermatology using mobile phone application served as a fast and accurate triage system and provided effective skin cancer care during the first wave of the COVID-19 pandemic. Teledermatology contributed to the reduction of the burden of the health system by minimizing outpatient visits and to decrease the risk of acquiring COVID-19 infection.
Acknowledgments
We thank Nóra Talmaci and Mercédesz Mazán for recording data in the clinical database prior to teledermatology examinations thus reducing the burden of doctors and assistants. Special thanks to Lili Róbert, Béla Háromszéki, Csilla Győrbíró, Kincső Blága, Anikó Kovács, Márta Medvecz, Bernadett Hídvégi, Ivett Dobribán-Ónodi, Alexandra Brunner, Csilla Haász, Andrea Gál, Anita Mohos, Tünde Kerner, Franciska Somogyi, László Fazekas, Pálma Silló, Martina Kádas, Andrea Litskay, Péter Bognár, Melinda Fábián, József Szakonyi, Dóra Faluhelyi, Anna Görög, Sarolta Kárpáti, Hunor Gergely, Béla Tamási, Dóra Beke, Krisztina Becker, Béla Tóth, Adrienn Poór, Veronika Tóth, and Klaudia Preisz, who took part in the teledermatology care during the first wave of the COVID-19 pandemic. We greatly appreciate the contribution of Veronika Orbán, Dóra Török, and Éva Fekete-Tóth who provided continuous IT support for dermatologists and patients, helped to identify and resolve the technical issues and other problems, performed ongoing administrative tasks and testing, and facilitated the organization and development of the teledermatology system.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph19052699/s1, Figure S1: Distribution of different triage groups among true positive (Panel A) and false negative diagnoses (Panel B) of other lesions, considering aggregated diagnosis of the lesions during teledermatology consultations.
Author Contributions
Conceptualization, A.J., N.K., F.A.M., L.F., K.L., N.M.W. and A.B.; methodology, A.J., N.K., F.A.M., K.F., S.B., Á.B. and A.B.; software, A.J. and P.S.; validation, A.J., F.A.M., K.F., D.P., Á.B., M.S. and A.B.; formal analysis, A.J. and Á.B.; investigation, A.J., F.A.M., K.F., D.P., L.F. and S.B.; resources, M.S. and P.S.; data curation, A.J., Á.B. and P.S.; writing—original draft preparation, A.J., N.K., N.M.W. and A.B.; writing—review and editing, A.J., N.K., F.A.M., K.F., D.P., S.B., L.F., E.S., K.L., M.S., N.M.W. and A.B.; visualization, A.J., N.K., F.A.M., K.F. and A.B.; supervision, N.K., K.L., E.S., M.S., N.M.W., P.S. and A.B.; project administration, P.S.; funding acquisition, M.S. and P.S. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by grants from the EFOP-3.6.3-VEKOP-16-2017-00009 (A.J.), ÚNKP-21-4-II-SE-10 (N.K.); New National Excellence Program of the Ministry for Innovation and Technology from the source of the National Research, Development and Innovation Fund of Hungary.
Institutional Review Board Statement
The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of National Institute of Pharmacy and Nutrition (OGYÉI/20793/2020).
Informed Consent Statement
Before uploading cases for teledermatology consultation via the mobile phone application, informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The data that support the findings of this study are available from the corresponding author A.B., upon reasonable request.
Conflicts of Interest
Péter Szoldán is co-owner and CEO of MedInnoScan Research and Development Ltd., which operates the store-and-forward teledermatology system described in this manuscript.
Footnotes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Valentino L.A., Skinner M.W., Pipe S.W. The role of telemedicine in the delivery of health care in the COVID-19 pandemic. Haemophilia. 2020;26:e230–e231. doi: 10.1111/hae.14044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Merkely B., Szabo A.J., Kosztin A., Berenyi E., Sebestyen A., Lengyel C., Merkely G., Karady J., Varkonyi I., Papp C., et al. Novel coronavirus epidemic in the Hungarian population, a cross-sectional nationwide survey to support the exit policy in Hungary. Geroscience. 2020;42:1063–1074. doi: 10.1007/s11357-020-00226-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bhaskar S., Nurtazina A., Mittoo S., Banach M., Weissert R. Editorial: Telemedicine During and Beyond COVID-19. Front. Public Health. 2021;9:662617. doi: 10.3389/fpubh.2021.662617. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Han E., Tan M.M.J., Turk E., Sridhar D., Leung G.M., Shibuya K., Asgari N., Oh J., García-Basteiro A.L., Hanefeld J., et al. Lessons learnt from easing COVID-19 restrictions: An analysis of countries and regions in Asia Pacific and Europe. Lancet. 2020;396:1525–1534. doi: 10.1016/S0140-6736(20)32007-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wosik J., Fudim M., Cameron B., Gellad Z.F., Cho A., Phinney D., Curtis S., Roman M., Poon E.G., Ferranti J., et al. Telehealth transformation: COVID-19 and the rise of virtual care. J. Am. Med. Inform. Assoc. 2020;27:957–962. doi: 10.1093/jamia/ocaa067. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Iyengar K., Mabrouk A., Jain V.K., Venkatesan A., Vaishya R. Learning opportunities from COVID-19 and future effects on health care system. Diabetes Metab. Syndr. 2020;14:943–946. doi: 10.1016/j.dsx.2020.06.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Hincapié M.A., Gallego J.C., Gempeler A., Piñeros J.A., Nasner D., Escobar M.F. Implementation and Usefulness of Telemedicine During the COVID-19 Pandemic: A Scoping Review. J Prim. Care Community Health. 2020;11:2150132720980612. doi: 10.1177/2150132720980612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Patel S.Y., Mehrotra A., Huskamp H.A., Uscher-Pines L., Ganguli I., Barnett M.L. Trends in Outpatient Care Delivery and Telemedicine During the COVID-19 Pandemic in the US. JAMA Intern. Med. 2021;181:388–391. doi: 10.1001/jamainternmed.2020.5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Tensen E., van der Heijden J.P., Jaspers M.W., Witkamp L. Two Decades of Teledermatology: Current Status and Integration in National Healthcare Systems. Curr. Dermatol. Rep. 2016;5:96–104. doi: 10.1007/s13671-016-0136-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Weinstein R.S., Krupinski E.A., Doarn C.R. Clinical Examination Component of Telemedicine, Telehealth, mHealth, and Connected Health Medical Practices. Med. Clin. North Am. 2018;102:533–544. doi: 10.1016/j.mcna.2018.01.002. [DOI] [PubMed] [Google Scholar]
- 11.North S. Telemedicine in the Time of COVID and Beyond. J. Adolesc. Health. 2020;67:145–146. doi: 10.1016/j.jadohealth.2020.05.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Ramirez A.V., Ojeaga M., Espinoza V., Hensler B., Honrubia V. Telemedicine in Minority and Socioeconomically Disadvantaged Communities Amidst COVID-19 Pandemic. Otolaryngol. Head Neck Surg. 2021;164:91–92. doi: 10.1177/0194599820947667. [DOI] [PubMed] [Google Scholar]
- 13.Rodriguez Socarrás M., Loeb S., Teoh J.Y., Ribal M.J., Bloemberg J., Catto J., N’Dow J., Van Poppel H., Gómez Rivas J. Telemedicine and Smart Working: Recommendations of the European Association of Urology. Eur. Urol. 2020;78:812–819. doi: 10.1016/j.eururo.2020.06.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Gillman-Wells C.C., Sankar T.K., Vadodaria S. COVID-19 Reducing the Risks: Telemedicine is the New Norm for Surgical Consultations and Communications. Aesthetic Plast. Surg. 2021;45:343–348. doi: 10.1007/s00266-020-01907-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Cantor J.H., McBain R.K., Pera M.F., Bravata D.M., Whaley C.M. Who Is (and Is Not) Receiving Telemedicine Care During the COVID-19 Pandemic. Am. J. Prev. Med. 2021;61:434–438. doi: 10.1016/j.amepre.2021.01.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Mehrtens S.H., Shall L., Halpern S.M. A 14-year review of a UK teledermatology service: Experience of over 40,000 teleconsultations. Clin. Exp. Dermatol. 2019;44:874–881. doi: 10.1111/ced.13928. [DOI] [PubMed] [Google Scholar]
- 17.Barros-Tornay R., Ferrándiz L., Martín-Gutiérrez F.J., Fernández-Orland A., Serrano-Gotarredona A., de la Torre J.M., Conejo-Mir M.D., Ojeda-Vila T., Márquez-Enríquez J., Hernández C., et al. Feasibility and cost of a telemedicine-based short-term plan for initial access in general dermatology in Andalusia, Spain. JAAD Int. 2021;4:52–57. doi: 10.1016/j.jdin.2021.05.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Moscarella E., Pasquali P., Cinotti E., Tognetti L., Argenziano G., Rubegni P. A survey on teledermatology use and doctors’ perception in times of COVID-19. J. Eur. Acad. Dermatol. Venereol. 2020;34:e772–e773. doi: 10.1111/jdv.16843. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Conforti C., Lallas A., Argenziano G., Dianzani C., Di Meo N., Giuffrida R., Kittler H., Malvehy J., Marghoob A.A., Soyer H.P., et al. Impact of the COVID-19 Pandemic on Dermatology Practice Worldwide: Results of a Survey Promoted by the International Dermoscopy Society (IDS) Dermatol. Pract. Concept. 2021;11:e2021153. doi: 10.5826/dpc.1101a153. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bhargava S., McKeever C., Kroumpouzos G. Impact of COVID-19 pandemic on dermatology practices: Results of a web-based, global survey. Int. J. Womens Dermatol. 2021;7:217–223. doi: 10.1016/j.ijwd.2020.09.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Giavina Bianchi M., Santos A., Cordioli E. Dermatologists’ perceptions on the utility and limitations of teledermatology after examining 55,000 lesions. J. Telemed. Telecare. 2021;27:166–173. doi: 10.1177/1357633X19864829. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Kazi R., Evankovich M.R., Liu R., Liu A., Moorhead A., Ferris L.K., Falo L.D., Jr., English J.C., 3rd Utilization of Asynchronous and Synchronous Teledermatology in a Large Health Care System During the COVID-19 Pandemic. Telemed. J. eHealth. 2021;27:771–777. doi: 10.1089/tmj.2020.0299. [DOI] [PubMed] [Google Scholar]
- 23.Brinker T.J., Hekler A., von Kalle C., Schadendorf D., Esser S., Berking C., Zacher M.T., Sondermann W., Grabe N., Steeb T., et al. Teledermatology: Comparison of Store-and-Forward Versus Live Interactive Video Conferencing. J. Med. Internet Res. 2018;20:e11871. doi: 10.2196/11871. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Armstrong A.W., Wu J., Kovarik C.L., Goldyne M.E., Oh D.H., McKoy K.C., Shippy A.M., Pak H.S. State of teledermatology programs in the United States. J. Am. Acad. Dermatol. 2012;67:939–944. doi: 10.1016/j.jaad.2012.02.019. [DOI] [PubMed] [Google Scholar]
- 25.Lowe A., Atwan A., Mills C. Teledermoscopy as a community based diagnostic test in the era of COVID-19? Clin. Exp. Dermatol. 2021;46:173–174. doi: 10.1111/ced.14399. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Teuscher M., Diehl K., Schaarschmidt M.L., Weilandt J., Sasama B., Ohletz J., Könnecke A., Harth W., Hillen U., Peitsch W.K. Effects of the COVID-19 pandemic on care of melanoma patients in Berlin, Germany: The Mela-COVID survey. Eur. J. Dermatol. 2021;31:521–529. doi: 10.1684/ejd.2021.4098. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Gomolin T., Cline A., Handler M.Z. The danger of neglecting melanoma during the COVID-19 pandemic. J. Dermatol. Treat. 2020;31:444–445. doi: 10.1080/09546634.2020.1762844. [DOI] [PubMed] [Google Scholar]
- 28.Marson J.W., Maner B.S., Harding T.P., Meisenheimer J.V., Solomon J.A., Leavitt M., Levin N.J., Dellavalle R., Brooks I., Rigel D.S. The magnitude of COVID-19’s effect on the timely management of melanoma and nonmelanoma skin cancers. J. Am. Acad. Dermatol. 2021;84:1100–1103. doi: 10.1016/j.jaad.2020.12.065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Hoellwerth M., Kaiser A., Emberger M., Brandlmaier M., Laimer M., Egger A., Bauer J.W., Koelblinger P. COVID-19-Induced Reduction in Primary Melanoma Diagnoses: Experience from a Dermatopathology Referral Center. J. Clin. Med. 2021;10:59. doi: 10.3390/jcm10184059. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Gisondi P., Cazzaniga S., Di Leo S., Piaserico S., Bellinato F., Pizzolato M., Gatti A., Eccher A., Brunelli M., Saraggi D., et al. Impact of the COVID-19 pandemic on melanoma diagnosis. J. Eur. Acad. Dermatol. Venereol. 2021;35:e714–e715. doi: 10.1111/jdv.17493. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Silveira C., Carcano C., Mauad E., Faleiros H., Filho A. Cell phone usefulness to improve the skin cancer screening: Preliminary results and critical analysis of mobile app development. Rural Remote Health. 2019;19:4895. doi: 10.22605/RRH4895. [DOI] [PubMed] [Google Scholar]
- 32.Chuchu N., Dinnes J., Takwoingi Y., Matin R.N., Bayliss S.E., Davenport C., Moreau J.F., Bassett O., Godfrey K., O’Sullivan C., et al. Teledermatology for diagnosing skin cancer in adults. Cochrane Database Syst. Rev. 2018;12:CD013193. doi: 10.1002/14651858.CD013193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Barbieri J.S., Nelson C.A., James W.D., Margolis D.J., Littman-Quinn R., Kovarik C.L., Rosenbach M. The reliability of teledermatology to triage inpatient dermatology consultations. JAMA Dermatol. 2014;150:419–424. doi: 10.1001/jamadermatol.2013.9517. [DOI] [PubMed] [Google Scholar]
- 34.Giavina-Bianchi M., Santos A.P., Cordioli E. Teledermatology reduces dermatology referrals and improves access to specialists. EClinicalMedicine. 2020;29–30:100641. doi: 10.1016/j.eclinm.2020.100641. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Warshaw E.M., Lederle F.A., Grill J.P., Gravely A.A., Bangerter A.K., Fortier L.A., Bohjanen K.A., Chen K., Lee P.K., Rabinovitz H.S., et al. Accuracy of teledermatology for pigmented neoplasms. J. Am. Acad. Dermatol. 2009;61:753–765. doi: 10.1016/j.jaad.2009.04.032. [DOI] [PubMed] [Google Scholar]
- 36.Moreno-Ramirez D., Ferrandiz L., Nieto-Garcia A., Carrasco R., Moreno-Alvarez P., Galdeano R., Bidegain E., Rios-Martin J.J., Camacho F.M. Store-and-forward teledermatology in skin cancer triage: Experience and evaluation of 2009 teleconsultations. Arch. Dermatol. 2007;143:479–484. doi: 10.1001/archderm.143.4.479. [DOI] [PubMed] [Google Scholar]
- 37.Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. doi: 10.2307/2529310. [DOI] [PubMed] [Google Scholar]
- 38.Yeboah C.B., Harvey N., Krishnan R., Lipoff J.B. The Impact of COVID-19 on Teledermatology: A Review. Dermatol. Clin. 2021;39:599–608. doi: 10.1016/j.det.2021.05.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Temiz S.A., Dursun R., Daye M., Ataseven A. Evaluation of dermatology consultations in the era of COVID-19. Dermatol. Ther. 2020;33:e13642. doi: 10.1111/dth.13642. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kravets K., Vasylenko O., Dranyk Z., Bogomolets O. Store-and-forward teledermatology for the most common skin neoplasms in Ukraine. Acta Derm. Alp. Pannonica Adriat. 2018;27:79–83. doi: 10.15570/actaapa.2018.18. [DOI] [PubMed] [Google Scholar]
- 41.Congalton A.T., Oakley A.M., Rademaker M., Bramley D., Martin R.C. Successful melanoma triage by a virtual lesion clinic (teledermatoscopy) J. Eur. Acad. Dermatol. Venereol. 2015;29:2423–2428. doi: 10.1111/jdv.13309. [DOI] [PubMed] [Google Scholar]
- 42.Massone C., Maak D., Hofmann-Wellenhof R., Soyer H.P., Frühauf J. Teledermatology for skin cancer prevention: An experience on 690 Austrian patients. J. Eur. Acad. Dermatol. Venereol. 2014;28:1103–1108. doi: 10.1111/jdv.12351. [DOI] [PubMed] [Google Scholar]
- 43.Singhal R.R., Talati K.N., Gandhi B.P., Shinde M.K., Nair P.A., Phatak A.G. Prevalence and Pattern of Skin Diseases in Tribal Villages of Gujarat: A Teledermatology Approach. Indian J. Community Med. 2020;45:199–203. doi: 10.4103/ijcm.IJCM_76_19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Gimeno-Vicente M., Alfaro-Rubio A., Gimeno-Carpio E. Teledermatology by WhatsApp in Valencia: Characteristics of Remote Consultation and Its Emotional Impact on the Dermatologist. Actas Dermosifiliogr. 2020;111:364–380. doi: 10.1016/j.ad.2019.10.003. [DOI] [PubMed] [Google Scholar]
- 45.Ncube B., Mars M., Scott R.E. The need for a telemedicine strategy for Botswana? A scoping review and situational assessment. BMC Health Serv. Res. 2020;20:794. doi: 10.1186/s12913-020-05653-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Jakhar D., Kaul S., Kaur I. WhatsApp messenger as a teledermatology tool during coronavirus disease (COVID-19): From bedside to phone-side. Clin. Exp. Dermatol. 2020;45:739–740. doi: 10.1111/ced.14227. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Kroemer S., Frühauf J., Campbell T.M., Massone C., Schwantzer G., Soyer H.P., Hofmann-Wellenhof R. Mobile teledermatology for skin tumour screening: Diagnostic accuracy of clinical and dermoscopic image tele-evaluation using cellular phones. Br. J. Dermatol. 2011;164:973–979. doi: 10.1111/j.1365-2133.2011.10208.x. [DOI] [PubMed] [Google Scholar]
- 48.Lamel S.A., Haldeman K.M., Ely H., Kovarik C.L., Pak H., Armstrong A.W. Application of mobile teledermatology for skin cancer screening. J. Am. Acad. Dermatol. 2012;67:576–581. doi: 10.1016/j.jaad.2011.11.957. [DOI] [PubMed] [Google Scholar]
- 49.Giavina-Bianchi M., Azevedo M.F.D., Sousa R.M., Cordioli E. Part II: Accuracy of Teledermatology in Skin Neoplasms. Front. Med. 2020;7:845. doi: 10.3389/fmed.2020.598903. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Finnane A., Dallest K., Janda M., Soyer H.P. Teledermatology for the Diagnosis and Management of Skin Cancer: A Systematic Review. JAMA Dermatol. 2017;153:319–327. doi: 10.1001/jamadermatol.2016.4361. [DOI] [PubMed] [Google Scholar]
- 51.Silveira C.E., Silva T.B., Fregnani J.H., da Costa Vieira R.A., Haikel R.L., Jr., Syrjänen K., Carvalho A.L., Mauad E.C. Digital photography in skin cancer screening by mobile units in remote areas of Brazil. BMC Dermatol. 2014;14:19. doi: 10.1186/s12895-014-0019-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Cotes M.E., Daugherty L.N., Sargen M.R., Chen S.C. Diagnostic accuracy of teledermatology for nonmelanoma skin cancer: Can patients be referred directly for surgical management? J. Am. Acad. Dermatol. 2021;85:464–466. doi: 10.1016/j.jaad.2017.09.029. [DOI] [PubMed] [Google Scholar]
- 53.Şenel E., Baba M., Durdu M. The contribution of teledermatoscopy to the diagnosis and management of non-melanocytic skin tumours. J. Telemed. Telecare. 2013;19:60–63. doi: 10.1177/1357633X12474961. [DOI] [PubMed] [Google Scholar]
- 54.Manahan M.N., Soyer H.P., Loescher L.J., Horsham C., Vagenas D., Whiteman D.C., Olsen C.M., Janda M. A pilot trial of mobile, patient-performed teledermoscopy. Br. J. Dermatol. 2015;172:1072–1080. doi: 10.1111/bjd.13550. [DOI] [PubMed] [Google Scholar]
- 55.Arzberger E., Curiel-Lewandrowski C., Blum A., Chubisov D., Oakley A., Rademaker M., Soyer H., Hofmann-Wellenhof R. Teledermoscopy in High-risk Melanoma Patients: A Comparative Study of Face-to-face and Teledermatology Visits. Acta Derm. Venereol. 2016;96:779–783. doi: 10.2340/00015555-2344. [DOI] [PubMed] [Google Scholar]
- 56.Knudsen K.E., Willman C., Winn R. Optimizing the Use of Telemedicine in Oncology Care: Postpandemic Opportunities. Clin. Cancer Res. 2021;27:933–936. doi: 10.1158/1078-0432.CCR-20-3758. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Hsiao J.L., Oh D.H. The impact of store-and-forward teledermatology on skin cancer diagnosis and treatment. J. Am. Acad. Dermatol. 2008;59:260–267. doi: 10.1016/j.jaad.2008.04.011. [DOI] [PubMed] [Google Scholar]
- 58.Moreno-Ramírez D., Romero-Aguilera G. Teledermatology: From the Tempest of Debate to Calmer Waters. Actas Dermosifiliogr. 2016;107:366–368. doi: 10.1016/j.ad.2016.01.006. [DOI] [PubMed] [Google Scholar]
- 59.Bozsányi S., Farkas K., Bánvölgyi A., Lőrincz K., Fésűs L., Anker P., Zakariás S., Jobbágy A., Lihacova I., Lihachev A., et al. Quantitative Multispectral Imaging Differentiates Melanoma from Seborrheic Keratosis. Diagnostics. 2021;11:1315. doi: 10.3390/diagnostics11081315. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Carrera C. The Many Faces of Seborrheic Keratosis. Actas Dermosifiliogr. 2019;110:338. doi: 10.1016/j.ad.2018.12.005. [DOI] [PubMed] [Google Scholar]
- 61.Moreno-Ramírez D., Raya-Maldonado J., Morales-Conde M., Ojeda-Vila T., Martín-Gutiérrez F.J., Ruíz-de-Casas A., Fernández-Orland A., Jm H.E., Ferrándiz L. Increasing Frequency of Seborrheic Keratosis Diagnoses as a Favorable Consequence of Teledermatology-Based Skin Cancer Screening: A Cross-sectional Study of 34,553 Patients. Am. J. Clin. Dermatol. 2017;18:681–685. doi: 10.1007/s40257-017-0283-z. [DOI] [PubMed] [Google Scholar]
- 62.Betlloch-Mas I., Martínez-Miravete M.T., Berbegal-DeGracia L., Sánchez-Vázquez L., Sánchez-Payá J. Teledermatology in paediatrics: Health-care impact on the early treatment of infantile haemangiomas. J. Telemed. Telecare. 2021;27:424–430. doi: 10.1177/1357633X20904901. [DOI] [PubMed] [Google Scholar]
- 63.Warshaw E.M., Gravely A.A., Nelson D.B. Reliability of store and forward teledermatology for skin neoplasms. J. Am. Acad. Dermatol. 2015;72:426–435. doi: 10.1016/j.jaad.2014.11.001. [DOI] [PubMed] [Google Scholar]
- 64.Clarke E.L., Reichenberg J.S., Ahmed A.M., Keeling B., Custer J., Rathouz P.J., Jambusaria-Pahlajani A. The utility of teledermatology in the evaluation of skin lesions. J. Telemed. Telecare. 2021:1–8. doi: 10.1177/1357633X20987423. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data that support the findings of this study are available from the corresponding author A.B., upon reasonable request.