Abstract
Introduction:
The number of newly diagnosed skin cancers per year is greater than the sum of the four most common cancers: breast, prostate, lung, and colon. The implementation of primary and secondary prevention measures, over the last 2 to 3 decades, has made a major contribution to successful treatment.
Aim:
Evaluate the accuracy and reliability of teledermoscopic versus clinical diagnosis for skin cancers when diagnostic algorithms are used, and when GPs and surgical specialties are involved in the clinical procedure.
Methods:
Digital dermoscope (TS-DD, by Teleskin company) was used for the acquisition of teledermoscopic photographs and specialized teledermoscopic software was used for clinical examination and teledermoscopic consultation. The teledermoscopic procedure itself was performed in two steps. The first step was a clinical examination using the ABCDE rule with digital dermoscopic photography of the suspected lesion. The second step was a 2-step dermoscopic evaluation using the second step ABCD algorithm for the second step. Accuracy and diagnostic reliability were calculated for: teledermoscopic diagnosis versus histopathological diagnosis; clinical diagnosis versus histopathological diagnosis and teledermoscopic diagnosis versus clinical diagnosis.
Results:
The study included 120 patients with 121 Pigmented Skin Lesions, of which 75 (62%) were benign and 46 (38%) were malignant lesions (6 melanomas and 40 NonMelanoma Skin Cancers). Diagnostic accuracy between teledermoscopic and histopathologic diagnosis was 90.91% and reliability k=0.81; between clinical and histopathological diagnosis the diagnostic accuracy was 82.64% and the reliability k=0.64 and between the clinical and teledermoscopic diagnosis the diagnostic accuracy was 81.82% and the reliability k=0.62.
Conclusion:
The achieved diagnostic accuracy between clinical and teledermoscopic diagnosis, when using diagnostic algorithms, establishes a feasible screening path for skin cancers and indicates that general practitioners and specialized surgeons may equally be involved in prevention.
Keywords: teledermoscopy , prevention , skin cancers
1. INTRODUCTION
Skin cancers (melanoma and non-melanoma skin cancers), as the most common form of human cancer, are a global problem. The number of newly diagnosed skin cancers per year is higher than the sum of the other four most common cancers: breast, prostate, lung, and colon ( 1 ).
Melanoma and non-melanoma skin cancers (basocellular and squamocellular skin cancers) are the most common forms of skin cancer. Melanoma is characterized by two important characteristics: a continuous increase in the incidence of the disease and a high rate of metastasis. On the other hand, basocellular skin cancer (as the most common form of non-melanoma skin cancers) is characterized by a high degree of recurrence and new-onset. Based on the facts: that the therapeutic potential for metastatic melanoma is poor, as well as highly functional and social morbidity in basal cell carcinoma, there is a clear need for adequate early detection methods that will have a positive effect on prognosis.
Dermoscopy has been established over the last three decades as a reliable diagnostic method for the early detection of melanoma and other skin cancers ( 2 ). On the other hand, the development of new technologies, and above all telemedicine, has made the dermoscopy more widely available to the patient. Compared to clinical diagnostics, dermoscopy significantly improves diagnostic accuracy for all skin cancers, and especially for melanoma ( 3 , 4 ).
The term „Teledermoscopy“ was established just over two decades ago when, first when, Piccolo 1999 ( 5 ) and then Brown 2000 ( 6 ) published their first experiences. They then used the term teledermatoscopy. Teledermoscopy itself is just one of the branches of telemedicine whose roots are far deeper.
Specifically, Thomas Bird coined the term telemedicine ( 7 ) in the 1970s, which means „healing at a distance“ (from Greek „tele“ and Latin „medicus“) ( 8 , 9 ). However, the facts still indicate that as early as the early 20th century, Villem Einthoven, a Dutch physiologist, developed the first electrocardiograph, using a wire galvanometer and telephone wires, to record electrical cardiac signals from patients at a distance of 1.5 km ( 7 , 10 ).
Telemedicine, and therefore teledermoscopy, is often associated with the term „Store-and-forward“, which means collecting and forwarding information for evaluation, usually electronically. The information itself is, for now, transmitted by „electronic mail, uploaded to a secure website, or using a private network“.
Depending on the author, the accuracy of clinical diagnosis for skin cancers (face to face or naked eye) ranges from 40% ( 11 ), 54% ( 12 ) to 71.0% ( 13 ) for sensitivity, or 71.8% ( 12 ), 81.0 ( 13 ) to 84.6% ( 11 ) for specificity. When using the ABCDE rule in clinical examination, the accuracy of clinical diagnosis depends on the number of criteria involved in the decision and ranges from 43-97% for sensitivity and 36-100% for specificity, i.e. 65% for sensitivity and 89% for specificity when 2 of 5 ABCDE criteria are used nonselectively ( 14 ).
The accuracy of dermoscopic diagnosis for skin cancers, depending on the algorithm used, ranges from 55%–79.2% ( 11 , 12 ) for sensitivity and 71.8% -89% ( 12 , 11 ) for specificity.
The reliability of teledermoscopic diagnosis was verified in studies that compared the results with a dermoscopic or clinical (face to face) diagnosis. In studies where teledermoscopic diagnoses are compared with dermoscopic diagnoses, diagnostic reliability ranges from 51% -92% for sensitivity and 62% specificity ( 15 - 18 ). At the same time as teledermoscopic diagnosis is compared with a clinical (face to face) diagnosis, the diagnostic reliability is 67% ( 15 ).
Rare are the studies where the reliability between clinical (face to face) and teledermoscopic diagnosis is analyzed in cases where clinical diagnosis is made by a GP and teledermoscopic diagnosis by a teledermoscopist, who in the vast majority of studies is a dermatologist. In one such study, Job P van der Heijden recorded reliability of k=0.41– 0.63 between teledermoscopic and histopathological diagnosis, k=0.55–0.73 between clinical (face to face) and teledermoscopic diagnosis, and k=0.90 between clinical and histopathological diagnosis ( 19 ).
2. AIM
Evaluate the accuracy and reliability of teledermoscopic diagnosis relative to clinical diagnosis for skin cancers when using diagnostic algorithms. Evaluate the accuracy and reliability of a clinical examination performed by non-dermatologists, such as general practitioners or surgical specialty physicians.
3. METHODS
Teledermoscopic process
Digital dermoscope (TS-DD, Teleskin) with a resolution of 2048 x 1920 pixels was used for the acquisition of teledermoscopic photographs and specialized teledermoscopic software was designed for clinical examination based on the International Dermoscopy Society (IDS) recommendations. The Teleskin Teledermoscopy Private Network (TTDPN), which was formed in 2009, was used to communicate with the teledermoscopist. The study included 6 physicians. Clinical examination was performed by 5 physicians: 3 general practitioners, 1 general surgeon, and 1 plastic surgeon. Only one teledermoscopist, a plastic surgeon with 10 years’ experience in teledermoscopy and over 20 years in dermoscopy, worked on the teledermoscopic response.
Before the start of the study, a training course of 6 hours was organized for physicians conducting the clinical examination with the acquisition of digital dermoscopic photography.
The course is designed to train physicians, on the one hand, for clinical diagnosis of skin cancer using the ABCDE clinical algorithm, and on the other, for the use of a teledermoscopic network and a teledermoscopic system (digital dermoscope and specialized teledermoscopic data acquisition software, digital dermoscopic photographs, formation of a teledermoscopic referral and receipt of a teledermoscopic report).
To enter and work in the TTDPN software, all physicians have a specially designed code.
After entering the TTDPN, clinicians who performed the clinical examination entered the following information: basic patient identification data, skin cancer risk factors, Fitzpatrick skin phototype determination data, an anatomical region of the skin lesion, and ABCDE clinical lesion data. After their acquisition, a digital dermoscopy of the skin lesion was performed. In most cases, it was a digital photograph. However, two or more digital dermoscopic photographs were taken: in the case of poor quality of the first photo or in cases where the size of the skin lesion exceeded 15 mm in diameter. A clinical decision on the skin cancer suspicious lesion was made based on the ABCDE clinical algorithm.
A skin lesion was declared suspicious if there were positive at least 2 of the following 3 criteria (A-asymmetry; C-color variation and E-evolution). After entering all the data and digital dermoscopy of the skin lesion itself, a teledermoscopic instruction is automatically formed, containing: basic patient information (age, gender, skin phototype and risk factors for skin cancers); ABCDE skin lesion clinical data and anatomical region; and digital photography of the skin lesion (at least one). There is no personal information on the referral that identifies the patient, such as name, surname, address and contact information (telephone, email, etc.). The instruction itself with all the data and digital dermoscopic photography is forwarded to the teledermoscopic consultation by TTDPN.
The teledermoscopist, after joining the telemedicine network, accessed specialized software for receiving teledermoscopic instructions. After opening the referral, he/she has access to all clinical examination data. A two-step dermoscopic evaluation of pigmented skin lesions is based on IDS recommendations using the first-step dermoscopic algorithm and one of four recommended second-step algorithms ( 20 ). In this study, the results of a two-step dermoscopic evaluation (a two-step dermoscopy procedure) were analyzed using the second step ABCD dermoscopic algorithm. After the dermoscopic evaluation is completed, an automatic teledermoscopic report is formed, containing all the data as well as the instructions, except the patient’s identification data (names and surnames). In the note, which is free text entry, the teledermoscopist must make a recommendation as to what to do next, and according to IDS recommendations. Possible tips in the recommendation are excision; diagnostic biopsy; preventive removal; follow-up at short intervals: at 3, 6, or 12 months; regular annual checkup and self-checkup.
Only patients with histopathological analysis were included for analysis in this study. The teledermoscopic response time is limited to 1 to 2 business days and the delivery time of the final finding to the patient, up to a maximum of 7 days, from the receipt of the teledermoscopic result.
Inclusion criteria
For all the physicians who participated in the study, the minimum criterion was a training course with training on the ABCDE rule, the use of a telemedicine network and a teledermoscopic system (hardware and software) and that they are not dermatologists. For skin lesions, the criteria were that they belonged to pigmented skin lesions (PSL) with a confirmed histopathological diagnosis after excision. For the algorithms, the criteria were: the ABCDE clinical rule for clinical examination and a two-step dermoscopic procedure using the ABCD rule in the second step. There were two conditions for patients: that they were over 20 years of age, and that they gave their consent to the teledermoscopic consultation and its use for research purposes. For digital dermoscopic photographs, the condition was of good quality so that clear teledermoscopic differentiation could be established.
Statistical analysis
For descriptive statistics and statistical methods, the analytical and statistical tool SPSS (Statistical Product and Service Solutions) version 20 was used, and for the graphical representation of Microsoft Excel 2010. Accuracy and reliability were calculated for all skin cancers together (Melanoma and NonMelanomaSkin Cancers). The gold standard for calculation was histopathological diagnosis except for the comparison of results between the clinical (ABCDE rule) and teledermoscopic diagnosis, where the latter was used for the gold standard.
Sensitivity and specificity tests were used to calculate accuracy. On the other hand, Kappa statistics ( 21 ) were used to calculate the reliability values. Accuracy and diagnostic reliability were calculated for: teledermoscopic diagnosis versus histopathological diagnosis; clinical diagnosis (using ABCDE rule) versus histopathological diagnosis of teledermoscopic diagnosis and teledermoscopic diagnosis versus clinical diagnosis (using ABCDE rule). Results of Kappa values were interpreted as 0-0.20 as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial and 0.81-1.0 as almost perfect agreement ( 22 ).
Ethical permission was not acquired for this study because all patients were examined (face to face) by a doctor who recommended further treatment.
4. RESULTS
Patient information
The study was conducted at the Teledermoscopy Center, ORS Plastic Surgery in Belgrade, Serbia.
Between January 2019 and January 2020, teledermoscopic consultation was performed for 680 PSL in 648 patients. The study excluded: 45 patients with 45 PSL who were referred for teledermoscopic consultation by 2 dermatologists; 471 patients with 502 PSL who had no data on excision and histopathological diagnosis; 6 patients under the age of 20 with 6 PSL and doing poor quality digital photography of 6 patients with 6 PSL.
After elimination of 528 patients with 559 PSL, 120 patients with 121 PSL were enrolled in the study: 115 (95.8%) patients had 1 PSL and 5 (4.2%) with 2 PSL each; 64 (53.3%) female and 56 (46.7%) male; patient age ranged from 25 to 76 years (average 52 years); 99 (82.5%) patients were Fitzpatrick skin type 3 (FST), 16 (13.3%) were FST 2, 4 (3.3%) were patients with FTZ 1; 1 patient (0.8%) with FST 4; and 2 (1.7%) patients had previous nonmelanocytic skin cancer (NMSC) and 7 (5.8%) with atypical mole syndrome. Of the 121 PSLs in total, 75 (62%) were benign lesions and 46 (38%) were malignant (6 melanomas and 40 NMSCs), Table 1 .
Table 1. Number of benign or malignant lesions included in the study .
| Histopathology diagnosis | Pigmented skin lesions | |||
|---|---|---|---|---|
| Number | % | |||
|
Benign
n=75 (62 %) |
Melanocytic naevi | Common Naevi | 44 | 36,4 |
| Dysplastic Naevus | 5 | 4,1 | ||
| Seborrhoeic keratosis | 21 | 17,4 | ||
| Vascular lesion | 3 | 2,5 | ||
| Dermatofibroma | 2 | 1,7 | ||
|
Malignant
n=46 (38 %) |
Melanoma | 6 | 5,0 | |
| Basal cell carcinoma | 35 | 28,9 | ||
| Aktinic keratosis | 2 | 1,7 | ||
| Bowen disease | 1 | 0,8 | ||
| Squamous cell carcinoma | 2 | 1,7 | ||
| Sum | 121 | 100,0 | ||
Figure 1. Digital dermoscopy (TS-DD, by Teleskin d.o.o. company, Serbia) for the acquisition of teledermoscopic photographs and specialized teledermoscopic software for clinical examination and teledermoscopic consultation.
Accuracy and reliability
In the differentiation of malignant from benign PSL, the diagnostic accuracy between teledermoscopic and histopathologic diagnosis was 90.91% and reliability k=0.81; between clinical and histopathological diagnosis the diagnostic accuracy was 82.64% and the reliability was k=0.64; and between clinical and teledermoscopic diagnosis the diagnostic accuracy was 81.82% and the reliability k=0.62, Table 2 .
Table 2. Diagnostic Accuracy and Realibility of the Clinical (ABCDE) and Teledermoscopical (ABCD) diagnosis .
| Diagnostic diferentiation | Diagnostic level | Statistics | |||
|---|---|---|---|---|---|
|
Sensitivity
(95% CI) |
Specificity
(95% CI) |
Accuracy
(95% CI |
Kappa
(95% CI) |
||
| Malignant Vs Benign |
Teledermoscopy
(2SDP)
Vs Histopathology |
89.13%
(76.43% to 96.38%) |
92%
(83.40% to 97.01%) |
90.91%
(84.32% to 95.37%) |
0.81
(0.700 to 0.916) |
|
Clinic
(ABCDE)
Vs Histopathology |
80.43%
(66.09°% to 90.64%) |
84%
(73.72% to 91.45%) |
82.64%
(74.70% to 88.92%) |
0.64
(0.496 to 0.777) |
|
|
Vs Clinic
(ABCDE)
Vs
Teledermoscopy (2SDP) |
78.72%
(64.34% to 89.30%) |
83.78%
(73.39% to 91.33%) |
81.82%
(73.78% to 88.24%) |
0.62
0.477 to 0.763 |
|
Abbrevation: 2SDP, a two-step dermoscopy procedure; ABCDE , ABCDE rule for skin cancers detection
5. DISCUSSION
Studies analyzing clinical and teledermoscopic accuracy in the prevention of skin cancer, involving GPs, are rare. However, in the available literature to the author, for the time being, there is no study where, in addition to GPs, surgical specialty physicians also participate, as well as using clinical and dermoscopic algorithms in the analysis.
In this study, a slightly higher score of sensitive clinical diagnosis (80.43%) was observed compared to 40% to 71% ( 11 , 12 , 13 ) in studies performed by general practitioners with or without the use of ABCD (E) rules. At the same time, the specificity score (84%) ranged from 71.8% to 84.6% ( 11 , 12 ).
Large variations in accuracy scores, and in particular sensitivity, are directly related to the methodologically inaccurate use of the ABCDE rule. Specifically, Thomas ( 14 ), in the most comprehensive and clearly methodologically defined study of 1998, pointed to the striking behavior of the results of the ABCDE rule concerning the number of criteria involved in decision-making. The most consistent results were obtained when using 2 or 3 criteria, and then the sensitivity values ranged from 89% to 66% and 65% to 80%, respectively. The comparative analysis of the results presented above ( 11 - 14 ) indicates that in this study a balanced score of sensitivity and specificity (80.43% Vs 84%) was achieved due to the precise (not random) ( 14 ) choice of attributes involved in decision making (A, C and E).
High scores on the sensitivity and specificity of dermoscopic diagnosis were achieved in this study (89.13% and 92%), compared to the results in studies published by Menzies ( 11 ) and Argenziano ( 12 ) (55%–79.2% for sensitivity and 71.8% -89 % ( 12 , 11 ) for specificity), can be explained by the expected progress made by the experience, expressed by a time distance of at least 10 years.
Figure 2. Dermoscopic images: A, Dysplastic Naevus; B, Melanoma in Situ and C, Basal Cell Carcinoma ( low grade-nodular type) .
A comparison of the results of the accuracy of the teledermoscopic diagnosis with the histopathological diagnosis indicates that a sensitivity value of 89.13% was reached in the range of published results ranging from 51% -92% ( 15 - 18 ). However, the achieved specificity value of 92% is significantly higher than the published value of 67% ( 15 ).
The reliability of teledermoscopic and clinical diagnosis concerning histopathological diagnosis, in studies involving general practitioners, ranges from: „Moderate to Substantial agreement“, (k=0.41–0.63) for teledermoscopic reliability and (k=0.55–0.73) for clinical reliability ( 19 ); and „Almost perfect agreement“ in comparison of clinical with teledermoscopic diagnosis (k=0.90) ( 19 ). In contrast to these results, in our study, the reliability of teledermoscopic diagnosis compared to histopathological diagnosis was raised to the level of „Almost perfect agreement“ (k=0.81), and the reliability of clinical diagnosis: concerning histopathological diagnosis (k=0.64) and in relation on teledermoscopic diagnosis (k=0.62) remained at the „Substantial agreement“ value. In analyzing these results, one should always keep in mind the differences brought about by the size of the sample analyzed, 7 skin cancers in the P van der Heijden study ( 19 ), compared to the 46 skin cancers in this study.
All the results achieved in this study are either in the range published or raised to a higher level of confidence, with more uniform values of sensitivity and specificity.
6. CONCLUSION
Achieved diagnostic accuracy and reliability indicates that skin cancers can also be successfully identified by GPs and surgeons,in addition to dermatologists. Adding to the already emphasized importance of training and quality of photography, this study also highlights the importance of a clearly defined methodology and use of teledermoscopic software that includes clinical and dermoscopic algorithms.
Acknowledgments:
Declaration of patient consent: The authors certify that they have obtained all appropriate patient consent forms
Author’s Contribution:
Each author gave a substantial contribution to the conception or design of the work and in the acquisition, analysis and interpretation of data for the work. Each author had a role in drafting the work and revising it critically for important intellectual content. Each author gave final approval of the version to be published and they agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Conflicts of interest:
Jadran Bandic is shareholder of Teleskin d.o.o. Serbia.
Financial support and sponsorship:
Nil.
REFERENCES
- 1. Australian Bureau of Statistics. 3303.0–Causes of Death, Australia . Canberra : Australian Bureau of Statistics ; 2013 . 2015 . [Google Scholar]
- 2. Bowling J , Argenziano G , Azenha A , Bandic J , Bergman R , Blum A , et al. Dermoscopy Key Points: Recommendations from the International Dermoscopy Society . Dermatology . 2007 ; 214 : 3 – 5 . doi: 10.1159/000096904. [DOI] [PubMed] [Google Scholar]
- 3. Argenziano G , Soyer HP . Dermoscopy of pigmented skin lesions – a valuable tool for early diagnosis of melanoma . Lancet Oncol . 2001 ; 2 : 443 – 449 . doi: 10.1016/s1470-2045(00)00422-8. [DOI] [PubMed] [Google Scholar]
- 4. Kittler H , Pehamberger H , Wolff K , Binder M. Diagnostic accuracy of dermoscopy . Lancet Oncol . 2002 ; 3 : 159 – 165 . doi: 10.1016/s1470-2045(02)00679-4. [DOI] [PubMed] [Google Scholar]
- 5. Piccolo D , Smolle J , Wolf IH , Peris K , Hofmann-Wellenhof R , Dell’Eva G , et al. Face-to-face diagnosis vs telediagnosis of pigmented skin tumors: a teledermoscopic study . Arch Dermatol . 1999 ; 135 : 1467 – 1471 . doi: 10.1001/archderm.135.12.1467. [DOI] [PubMed] [Google Scholar]
- 6. Braun RP , Meier M , Pelloni F , Ramelet AA , Schilling M , Tapernoux B , et al. Teledermatoscopy in Switzerland: a preliminary evaluation . J Am Acad Dermatol . 2000 ; 42 : 770 – 775 . doi: 10.1067/mjd.2000.103977. [DOI] [PubMed] [Google Scholar]
- 7. Strehle EM , Shabde N. One hundred years of telemedicine: does this new technology have a place in paediatrics? . Arch Dis Child Educ Pract Ed . 2006 ; 91 ( 12 ): 956 – 959 . doi: 10.1136/adc.2006.099622. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Wootton R , Craig J. , Patterson V. 2nd . London : Royal Society of Medicine ; 2006 . Introduction to Telemedicine . [Google Scholar]
- 9. Zundel, K . Telemedicine: history, applications, and impact on librarianship . Journal of the medical library association . 1996 ; 84 ( 1 ): 71 – 79 . [PMC free article] [PubMed] [Google Scholar]
- 10. Einthoven W. Le t´ el´ecardiogramme . Arch Int Physiol . 1906 ; 4 : 132 – 164 . [Google Scholar]
- 11. Menzies SW , Emery J , Staples M , Davies S , McAvoy B , Fletcher J , et al. Impact of dermoscopy and short-term sequential digital dermoscopy imaging for the management of pigmented lesions in primary care: a sequential intervention trial . Br J Dermatol . 2009 ; 161 ( 6 ): 1270 – 1277 . doi: 10.1111/j.1365-2133.2009.09374.x. [DOI] [PubMed] [Google Scholar]
- 12. Argenziano G , Puig S , Zalaudek I , Sera F , Corona R , Alsina M , et al. Dermoscopy improves accuracy of primary care physicians to triage lesions suggestive of skin cancer . J Clin Oncol . 2006 ; 24 ( 12 ): 1877 – 1882 . doi: 10.1200/JCO.2005.05.0864. [DOI] [PubMed] [Google Scholar]
- 13. Vestergaard ME , Macaskill P , Holt PE , Menzies SW . Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: a meta-analysis of studies performed in a clinical setting . Br J Dermatol . 2008 ; 159 ( 3 ): 669 – 676 . doi: 10.1111/j.1365-2133.2008.08713.x. [DOI] [PubMed] [Google Scholar]
- 14. Thomas L , Tranchand P , Berard F , Secchi T , Colin C , Moulin G. Semiological value of ABCDE criteria in the diagnosis of cutaneous pigmented tumors . Dermatology . 1998 ; 197 : 11 – 17 . doi: 10.1159/000017969. [DOI] [PubMed] [Google Scholar]
- 15. Borve A , Terstappen K , Sandberg C , Paoli J. Mobile teledermoscopy-there’s an app for that . Dermatol Pract Concept . 2013 ; 3 ( 2 ): 41 – 48 . doi: 10.5826/dpc.0302a05. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Börve A , Dahlén Gyllencreutz J , TerstappenK , Sandberg C , Paoli J. Smartphone teledermoscopy referrals: a novel process for improved triage of skin cancer patients . Acta Derm Venereol . 2015 ; 95 ( 2 ): 186 – 190 . doi: 10.2340/00015555-1906. [DOI] [PubMed] [Google Scholar]
- 17. Congalton AT , Oakley AM , Rademaker M , Bramley D , Martin RCW . Successful melanoma triage by avirtual lesion clinic (teledermatoscopy) . J Eur Acad Dermatol Venereol . 2015 ; 29 ( 12 ): 2423 – 2428 . doi: 10.1111/jdv.13309. [DOI] [PubMed] [Google Scholar]
- 18. Senel E , Sabancılar E , Mansuroğlu C , Demir E. A preliminary study of the contribution of telemicroscopy to the diagnosis and management of skin tumours in teledermatology . J Telemed Telecare . 2014 ; 20 ( 4 ): 178 – 183 . doi: 10.1177/1357633X14533885. [DOI] [PubMed] [Google Scholar]
- 19. Heijden JP , Thijssing L , Witkamp L , Spuls PI , Keizer NF . Accuracy and reliability of teledermatoscopy with images taken by general practitioners during everyday practice . J Telemed Telecare . 2013 October ; 19 ( 6 ): 320 – 325 . doi: 10.1177/1357633X13503437. [DOI] [PubMed] [Google Scholar]
- 20. Argenziano G , Soyer HP , Chimenti S , Talamini R , Corona R , MD , Sera F , et al. Dermoscopy of pigmented skin lesions: results of a consensus meeting via the Internet . J Am Acad Dermatol . 2003 ; 48 ( 5 ): 679 – 693 . doi: 10.1067/mjd.2003.281. [DOI] [PubMed] [Google Scholar]
- 21. Viera AJ , Garrett JM . Understanding interobserver agreement: the kappa statistic . Fam Med . 2005 ; 37 : 360 – 363 . [PubMed] [Google Scholar]
- 22. Koch GG , Landis JR , Freeman JL , Freeman DH , Jr , Lehnen RC . A general methodology for the analysis of experiments with repeated measurement of categorical data . Biometrics . 1977 ; 33 : 133 – 158 . [PubMed] [Google Scholar]