Can We Trust the Use of Smartphone Cameras in Clinical Practice? Laypeople Assessment of Their Image Quality

Abstract

Background: Smartphone cameras are rapidly being introduced in medical practice, among other devices for image-based teleconsultation. Little is known, however, about the actual quality of the images taken, which is the object of this study. Materials and Methods: A series of nonclinical objects (from three broad categories) were photographed by a professional photographer using three smartphones (iPhone^® 4 [Apple, Cupertino, CA], Samsung [Suwon, Korea] Galaxy S2, and BlackBerry^® 9800 [BlackBerry Ltd., Waterloo, ON, Canada]) and a digital camera (Canon [Tokyo, Japan] Mark II). In a Web survey a convenience sample of 60 laypeople “blind” to the types of camera assessed the quality of the photographs, individually and best overall. We then measured how each camera scored by object category and as a whole and whether a camera ranked best using a Mann–Whitney U test for 2×2 comparisons. Results: There were wide variations between and within categories in the quality assessments for all four cameras. The iPhone had the highest proportion of images individually evaluated as good, and it also ranked best for more objects compared with other cameras, including the digital one. The ratings of the Samsung or the BlackBerry smartphone did not significantly differ from those of the digital camera. Conclusions: Whereas one smartphone camera ranked best more often, all three smartphones obtained results at least as good as those of the digital camera. Smartphone cameras can be a substitute for digital cameras for the purposes of medical teleconsulation.

Introduction

In 2012, over 6 billion people had access to mobile phones, with 90 mobile phone subscriptions per 100 people throughout the world¹ and a rapid penetration not only in high- but also in low- and middle-income countries. At the same time, over 90% of the worldwide population was covered by a mobile cellular network.¹ This, in turn, facilitated remarkable developments in information and communication technologies,² not least in the medical field, where access to clinical expertise of various kinds from various points of care was facilitated.

Smartphones, with additional features such as cameras, which are of great relevance for clinical support, have a penetration rate that is even faster than mobile phones. They are used heavily on an individual and daily basis among a variety of medical practitioners,^3,4 in particular for photographic images,⁵ and they are also promoted by institutions.^6,7

Studies have shown that image-based consultation of remote experts is possible in several specialties like dermatology, plastic surgery, or burn care.^8–12 However, the evidence rests predominantly on studies where photographs were taken with a digital camera, and only a few were conducted using older models of mobile phones.^13–17

Against this background it appears necessary to take a closer look at the quality of pictures taken with smartphone cameras in order to determine the value of incorporating smartphones in medical practice. In this study laypeople assess the quality of images taken by widely used smartphones and address the following questions: (1) Which cameras are regarded as providing images of good quality? (2) Which camera is considered the best?

Materials and Methods

Choice of Smartphones and Control Camera

We focused our investigation on current, but not the latest, generations of widely used smartphone models on the grounds that they are more likely to be in use in resource-poor settings where expert consultation may be more often required. Also, these smartphones would provide us with “bottom line” results given that smartphone cameras are continually improving. In addition to representing different operating systems, the smartphone cameras tested have varying technical features (shown in parentheses after the specific model). Three different smartphones in use in 2010–2012 were selected: an iPhone^® 4 (5 MP, 2592×1936 pixels, autofocus, light-emitting diode flash, and screen density of 330 pixels/inch [PPI]) (Apple, Cupertino, CA), a Samsung (Suwon, Korea) Galaxy S2 (8 MP, 3264×2448 pixels, autofocus, light-emitting diode flash, screen density of 218 PPI) (Android™ [Google, Mountain View, CA] platform), and a BlackBerry^® Torch 9800 (5 MP, 2592×1944 pixels, autofocus, light-emitting diode flash, screen density of 186 PPI) (BlackBerry Ltd., Waterloo, ON, Canada), as well as a professional digital camera with a 35-mm lens (Canon [Tokyo, Japan] Mark II) as a reference (“gold standard”).

Selection of Photographs

To avoid mixing clinical accuracy with image quality in the study, nonclinical objects from three broad categories were photographed for this specific study: nature, specific objects, and parts of objects. The objects were selected and photographed by a professional photographer, keeping in mind that they should be “neutral.” For each object, all photographs were taken during daylight, without a flash, in direct sequence but with random order of the cameras. As cameras vary from one smartphone to another, including comparable amount of information in a photograph was more important than having a set distance between a camera and the object. The final images were used in their raw state, without digital processing or retouching.

Of the 30 sets of pictures taken by the photographer, 15 were randomly selected and included in our Web survey using SurveyMonkey^® (www.surveymonkey.com/). They can be found in Figure 1.

Fig. 1.

Pictures of the 15 objects presented in the survey as taken with the Canon professional camera: (A–C) nature, (D–J) specific objects or subjects, or (K–O) parts of objects.

Data Collection Instrument and Procedure

The survey functioned as follows. For each picture the four photographs were shown simultaneously on a computer screen in random order, and the type of camera was not shared with the participants. The participants had two tasks for each picture: (1) assess the quality of each photograph on a 5-point Likert scale (excellent, good, borderline, poor, and noninterpretable) and (2) determine which of the four photographs was the best one (forced-choice methodology).

Furthermore, at the end of the survey, the participants were asked to rank in order, from most important (ranked first) to least important (ranked fifth), the following five features of image quality: focus, resolution, contrast, color, and composition.

Surveys were completed individually on one of two identical laptop computers (Dell [Round Rock, TX] Vostro i3 2.4 GHz with 3 GB RAM Windows 7 [Microsoft, Redmond, WA]) with a 15.4-inch screen, both set to a screen resolution of 1366×768 pixels and screen density of 101 PPI in a large room with low lighting. A research group member introduced the survey to each participant individually and was present in the room during survey completion.

The participants had as much time as desired to go through the set of photographs (average duration of approximately 30 min). They could zoom in or out of the screen in order to see one picture specifically or all four at the same time.

Participants

Assessment by laypeople is acknowledged as the most reliable tool to assess image quality.^18,19 Thus, 60 participants were recruited in two academic settings using convenience sampling among students and staff linked to the co-authors' affiliations. There were 20 participants recruited in South Africa who completed the survey between November 2013 and March 2014 and an additional 40 in Sweden who conducted the survey between March and April 2014. The mean age of the participants was 33 years old, ranging from 19 to 65 years old, and 52% were females.

Data Treatment

To determine if the pictures of any given camera were considered as of good quality, we retained ratings as both good and excellent (1 and 2 on the Likert scale). For each picture, we expressed as a percentage the number of times the participants rated the picture of a given camera as good/excellent ([n/60]×100). Those camera-specific percentages were then “averaged,” and all pictures were aggregated (15 pictures), as well as by category of pictures (3, 7, or 5 pictures). Furthermore, to test whether there was a significant difference in the overall or category-specific percentages, we compared the cameras 2×2 using the nonparametric Mann–Whitney U test (significance level set at p<0.05).

To determine whether one camera was “best,” we retrieved for each specific picture the number of times one camera was ranked as “best” (out of 60 responses per picture). We then compiled for each camera the number of “best” ratings across all pictures—and by category of pictures—and expressed the results in percentages (the sum of camera percentages adding up to 100%). As for the previous analyses, to test for significance in any difference in percentages, we compared 2×2 the camera-specific share of “best” using the nonparametric Mann–Whitney U test (significance level set at p<0.05).

We finally determined the rank of importance of each of the five image features presented to the participants by compiling, for each of them, the percentage of times they were ranked from most to least important (from 1 to 5).

Results

Figure 2 presents the distribution of the number of ratings of good or excellent quality expressed as percentages. There were great variations in the ratings both within cameras between pictures and between cameras for a given picture. The variation between pictures was greatest for the Canon digital camera and smallest for the iPhone, with more answers indicating good/excellent quality; the Samsung and BlackBerry had similar ranges of answers. The 2×2 analysis showed statistical significant differences only between the iPhone and the other cameras. There were no significant differences found when the Canon digital camera was compared with the Samsung or with the BlackBerry.

Fig. 2.

Camera-specific percentages of a good or excellent quality rating for each picture. Minimum, maximum, and median values are presented. The 2×2 comparisons using the Mann–Whitney U test revealed three significant differences, all with the iPhone: *p<0.05, **p<0.01.

Figure 3 presents the results for the number of times a given camera was ranked “best.” Across all pictures the iPhone ranked best most often, followed by the Canon. There were no statistically significant differences between the Canon and any of the smartphones; however, the iPhone was chosen significantly more often than the Samsung and BlackBerry.

Fig. 3.

Camera-specific number of times a picture was rated best across all pictures, expressed as a percentage (blind assessment). The 2×2 comparisons using the Mann–Whitney U test revealed two significant differences, both with the iPhone: **p<0.01.

Table 1 presents in percentages the mean number of answers that rated each picture as of good/excellent quality, and in parentheses is given the mean number of times each photograph was chosen as best for each category of pictures. In most cases a high rating by the participants for a given photograph was associated with a higher percentage of participants defining this photograph as the best one. Overall, apart from the iPhone, for which 72.7% of the pictures were rated as good or excellent, the other three cameras were only rated about half of the time as taking pictures of good or excellent quality. These lower ratings could not be explained based on the categories of the pictures as no significant differences could be seen between them for a given camera.

Table 1.

Camera-Specific Performance in Absolute and Relative Terms

	CANON		IPHONE		SAMSUNG		BLACKBERRY		ALL CAMERAS
	A	R	A	R	A	R	A	R	A	R
Nature pictures	64.5	(30.0)	82.8	(51.7)	43.3	(8.3)	53.3	(10.0)	62.5	(100.0)
Specific objects	55.5	(29.1)	65.5	(30.5)	64.5	(23.8)	50.9	(16.7)	59.1	(100.0)
Parts of objects	43.3	(24.0)	76.6	(44.7)	42.3	(15.0)	56.3	(16.3)	55.7	(100.0)
All pictures	53.2	(27.6)	72.7	(39.5)	52.9	(17.8)	53.2	(15.2)	58.0	(100.0)

A, absolute terms (average percentage of answers giving a good or excellent quality rating); R, relative terms (average number of times considered as best).

Looking at the ranking (from 1 to 5) of each of the five image quality features, it appears that over one out of three participants ranked “focus” most often as the most important criterion (35.6%), followed by resolution (26.3%) and “composition” as least important by 64.8% of participants (Fig. 4).

Fig. 4.

Percentage of time an image quality feature ranked from 1 (most important) to 5 (less important).

Discussion

Studies indicate that diagnostic accuracy is closely linked to good image quality,^20,21 which calls for critical evaluation of the tools most likely to be used by medical professionals (i.e., smartphones).^5,6,22 This study highlights several findings of interest in that respect. To begin with there is a great variation in the perceived quality of the pictures taken by any given camera, within and across picture categories. Furthermore, for any single object photographed all three smartphone cameras compare well with the digital one. Nonetheless, the iPhone stands out to the extent that its pictures are perceived significantly better than that of the digital camera, although both were chosen as best approximately the same number of times.

To the best of our knowledge this is the first study to compare the perceived quality of pictures taken with smartphones versus a digital camera used as a comparison (“gold standard”). Some earlier studies compared several digital cameras available at a point in time among each other in order to choose the best one for telemedicine purposes.^23,24 These evaluations consisted of a range of items, including but not limited to image quality. Whereas the image quality of several cameras was satisfactory, the authors' final decision was based on many other aspects.²⁴

A surprising result in our study is that the Canon professional digital camera, which was used as the reference, was not assessed as being the best performer. The characteristics of neither the participants nor the pictures explained these results. In fact, no significant differences could be seen between categories of pictures, indicating that the quality of the photographs did not depend on the item on the picture specifically. The information provided by the participants indicates that, in making their judgment on the quality of a picture, focus and resolution, two features related to fine details that can be seen in an image, together with color, are most important for several of them. It is beyond the scope of this study to extrapolate this information to that of an expert teleconsultation, but one can imagine that focus and resolution are of important in the medical field as well, for instance, in diagnostic assistance.

The study has several methodological strengths that can be stressed. Building on the perception of laypeople is acknowledged as the most reliable tool to assess image quality.^18,19 We also made sure that both the participants and the photographs chosen were “neutral,” with no clinical connotation, in order to not mix the notions of image quality and diagnosability. All pictures were taken by a professional photographer, which safeguarded an optimal image capture. It is of note, however, that the use of flash has been recommended for telediagnosis purposes when using digital cameras.^25–27 It is unclear whether this would also be recommended with a smartphone camera and whether the use of flash would have yielded different results from the results obtained here without flash.²⁸

Our results provide good indications that smartphones from three different platforms (Apple, BlackBerry, and Android) can provide medical practitioners with a working tool that can satisfactorily assist photographic documentation and image-based communication. New generations of all three models used are available and likely to perform at least as well as the ones we have studied, and, hopefully, costs barriers may be reduced in the future so that these smartphones can be available to professionals from resource-poor settings.

Our results are silent regarding what would happen on the part of the tele-expert should he or she be using any of those devices for the consultation of images sent to him or her by others for consultation, which also is an important issue. What we know from earlier studies is that images sent for teleconsultation (e.g., in radiology, dermatology, or plastic surgery) are often studied on a computer screen.⁹ However, there are good reasons to believe that there will be an increasing volume of advice provided using either smartphones or tablets. Studies are therefore needed that address this more closely.

Conclusions

Although one smartphone camera ranked best more often, all three smartphones obtained results at least as good as the digital camera. Smartphone cameras can be a substitute for digital cameras for the purposes of medical teleconsulation.

Acknowledgments

This study was supported by the Swedish International Development Agency and the Marianne and Marcus Wallenberg Foundation.

Disclosure Statement

No competing financial interests exist.