Analysis of the inter- and intra-observer agreement in radiographic evaluation of wrist fractures using the multimedia messaging service

Abstract

Background

Orthopaedic surgeons are often asked to evaluate X-rays of patients admitted to the Accident and Emergency Department with the suspicion of a wrist fracture or, in the case of an evident fracture, to decide the correct treatment. The aim of this study was to evaluate the feasibility of a correct interpretation of the images of injured wrists on the screen of a last generation mobile phone, in order to evaluate if the specialist could make the right diagnosis and choose the correct treatment.

Methods

Five orthopaedic and one hand surgeons have evaluate the X-rays of 67 patients who sustained an injury to their wrist. In the case of fracture, they were asked to classify it according to the AO and Mayo classification systems. The evaluation of the images was accomplished through the PACS and using a mobile phone, at a different time. In order to check the inter- and intra-observer reliability, the same pattern was followed after a few months.

Results

The mobile phone showed basically the same agreement between the observers highlighting the worsening of the inter- and intra-observer reliability with the increment of the variables considered by a classification system.

Conclusions

The present paper confirms that a last generation mobile phone can already be used in the clinical practise of orthopaedic surgeons on call who could use it as a useful device in remote or poorly served areas for a rapid and economic consultation

Level of Evidence

The level of evidence of this case is economic and decision analysis, level 2

Introduction

From the pioneering applications in space exploration and from the selective use in remote and poorly served areas [5–7], telemedicine has rapidly extended to Medicine [1, 6–8, 16–20, 25, 26, 28, 32, 37–41] even though not always cost-effectively, because of the expensive technology. The arrival of third generation mobile phones has allowed to lower costs and open new potentials for telemedicine [3, 9, 19, 20, 23, 26, 28, 30, 34, 40].

It is our experience that the verbal description of the radiographs of a musculoskeletal limb injury can be inaccurate, and therefore no substitute for viewing the actual images. In our hospital senior surgeons on call have access to all X-ray imaging from home thanks to an Internet-based connection to the hospital picture archiving and communication system (PACS). Multimedia messaging service (MMS) can also be used to transmit radiological images to experienced staff to permit accurate and prompt diagnosis and commence optimal treatment, still complying with costs [3] and medico-legal issues [22]. Starting from recent feasibility studies [3, 9, 19, 20, 23, 26, 28, 30, 34, 38–40], we have sought an alternative solution to the Internet based connection to the hospital PACS. This would be of use not only to doctors not connected to the Web and practising in remote areas of our country but also to hospital specialists, relieving them from the constant dependence on an Internet connection. The purpose of this study is to analyse the inter-observer and intra-observer agreement in the assessment of standard radiographs of distal radius fractures viewed either on a PACS station or on a mobile phone after transmission by MMS.

Methods

This is a retrospective study of 67 consecutive wrist radiographs of different patients (M = 17 F = 50; mean age, 64 years; range, 18–82) after a hyperextension wrist injury, who attended our Accident and Emergency Department between January and December 2008.

The patients included in the study received a standard radiographic assessment with a complete description of the clinical findings on the x-ray form in order to help the consultant radiologist in providing a precise diagnosis.

Among the 67 patients, 58 sustained a fracture of the distal radius with or without an associated fracture of the distal ulna. Standard A-P and lateral radiographs were blinded, randomized and later analysed by three experienced Trauma surgeons (GG, GR, MF) and a hand surgeon (FC). The analysis was done first on a PACS station and then, after 3 weeks, on a mobile phone commonly used when on call (HTC 3,470, 2-megapixel camera, 320 × 240 screen resolution). The images sent over the phone had been taken with the phone camera pointing directly to the PACS screen, avoiding all the patients’ identifiable details.

It is worth mentioning that we were required to maintain some distance between the PDA phone and the PACS monitor when capturing a displayed image because of the influence of reflected light. This could have affected the image quality.

A second analysis of the same pictures was carried out after 2 months, this time starting with the MMS images followed, after 3 weeks, by a viewing on the PACS screen. No additional manipulation such as moving or zooming was permitted.

Each observer was asked to confirm or exclude a fracture of the distal radius and/or the ulnar styloid and, if present, to classify the fracture according to the AO (type and group) [31] and Mayo [12] classifications. The observers had available for every viewing the original classification diagrams. They were also given the choice to judge the quality of each image as acceptable or non acceptable according to Archbold [3].

The analysis of the inter- and intra-observer agreement on diagnosis and classification was carried out by the first author (AF) who was not involved in the image viewing. It is based on the calculation of κ as set out by Cohen and Fleiss [10, 13, 14]. The degree of agreement is based on the Landis and Koch’s table [27] which provides an arbitrary subdivision of the κ coefficient ranging from no agreement (κ = 0) to perfect agreement (κ = 1; Table 1). The authors have not received any kind of funding by public or private companies.

Table 1.

κ value and corresponding degree of agreement [27]

κ value	Agreement
<0	Poor
0.0–0.20	Slight
0.21–0.40	Fair
0.41–0.60	Moderate
0.61–0.80	Substantial
0.81–1.00	Almost perfect

Results

The agreement values are shown in Tables 2 and 3. No images were judged unacceptable.

Table 2.

Inter-observer agreement

Observers (4)	κ value
	PDA		PACS
	First look	Second look	First look	Second look
Radius fracture	0.93	1	0.90	1
Ulnar styloid fracture	0.75	0.68	0.75	0.76
A.O. classification
Types	0.64	0.65	0.62	0.68
Groups	0.48	0.52	0.49	0.50
Mayo classification	0.61	0.63	0.66	0.65

PACS picture archiving and communication systems, PDA personal digital assistant

Table 3.

Intra-observer agreement

	κ value
	Observers	Radius fracture	Ulnar styloid fracture	AO		Mayo
				Types	Groups
PACS	4	1	0.76	0.7	0.55	0.58
PDA	4	0.93	0.75	0.59	0.50	0.57

PACS picture archiving and communication systems, PDA personal digital assistant

Inter-observer Agreement

The level of agreement in detecting the presence or formal exclusion of a fracture of the distal radius was almost perfect both on the mobile phone and on the PACS station (κ = 0.93 for mobile and κ = 1 for PACS). Agreement in spotting or excluding a fracture of the ulnar styloid was good (κ = 0.76 for mobile and κ = 0.75 for PACS).

Agreement in the AO classification was good (PACS κ = 0.68, mobile κ = 0.64) whereas it went down to moderate level for subgroups (PACS κ = 0.50, mobile κ = 0.49).

Agreement in the Mayo classification was good (PACS κ = 0.65, mobile κ = 0.61) without significant differences either between the two viewing methods or compared to the values for the AO classification. The agreement values of the second reading after 2 months were in average higher than those of the initial reading but there was no change of class in the Landis and Koch table.

Intra-observer Agreement

The agreement in detecting or excluding a fracture was almost perfect for both viewing methods for the radius (PACS κ = 1, mobile κ = 0.93). It was only good for the ulnar styloid (PACS κ = 0.76, mobile κ = 0.75).

The agreement in the AO classification was slightly better for the PACS (κ = 0.7) than for the mobile (κ = 0.59). There were no substantial differences for the classification into subgroups (PACS κ = 0.55, mobile κ = 0.50) and for the Mayo classification (PACS κ = 0.58, mobile κ = 0.57).

Inter-observer Agreement of PACS vs Phone

In order to further assess the potentiality of the mobile phone to read an X-ray without the PACS support, we have evaluated if the agreement between the readings of the two systems by different viewers was statistically significant within a 95% confidence interval.

The κ value in detecting or excluding a fracture to the distal radius has been 0.95 with a significant p value of 0.044.

The p value dropped to a non-significant level for the detection of a fracture to the ulnar styloid (κ value = 0.8, p value = 0.069), the AO type (κ value = 0.65, p value = 0.066), the AO group (κ value = 0.55, p value = 0.35) and the Mayo classification (inter-observer agreement = 0.57, p value = 0.06).

The cell phone, when compared to the radiologic report, has shown a specificity of 81% and a sensitivity of 89% in identifying a fracture to the radius. Both the values dropped when evaluating a potential fracture to the ulnar styloid. In the latter case, the specificity scored 81% and the sensitivity just 77%.

Discussion

The primary goal of this study was to assess the reliability of the MMS radiographic evaluation of potential wrist fracture. We correctly hypothesised that the inter and intra-observer agreement in radiographic diagnosis and classification of wrist fracture for MMS and PACS would be similar even though the inter-observer agreement does not necessarily imply diagnostic accuracy [9].

The number of variables in a classification affects the level of agreement in conventional radiography [2, 24, 29]. The highest agreement in our study was achieved in the analysis with two variables (detection/exclusion of a fracture) for the radius which has been the only case to show a p value < 0.05 when comparing directly the readings of the PACS and mobile phone. The worse agreement, and p value for the ulnar styloid, is probably due to its smaller size and more difficult visualisation on a lateral view.

The addition of another type of classification with multiple variables increases the disagreement between viewers [2, 21, 24, 29, 33, 35, 36] and this effect was confirmed in our study by a general decline of the level of agreement and p value. Nonetheless, viewing on a PACS station, with its high screen resolution, did not produce a better agreement than on a mobile phone. This shows that more sophisticated tools do not necessarily lead to better agreement between viewers [15].

Our agreement values are similar to those in the literature (Tables 4 and 5; [35–39]) and they show that agreement depends neither on the mode of capture nor on transmission and reproduction of the images. Our wrist radiographs could be viewed with the same diagnostic agreement on a PACS screen and on the screen of a mobile phone to which they had been sent by MMS.

Table 4.

Comparision of inter-observer agreement in the literature

	κ value
	Observers	Radius fracture	Ulnar styloid fracture	AO		Mayo
				Types	Groups
Bellinzona PACS	4	0.90	0.75	0.62	0.49	0.66
Bellinzona PDA	4	0.93	0.75	0.64	0.48	0.61
Jin et al. [29]	5			0.45–0.48 (0.28–0.71)	0.25–0.29 (0.18–0.37)
Oskam et al. [14]	2			0.65–0.86
Olivera Filho et al. [12]	9				0.21 (with subgroups)
Andersen et al. [31]	4		0.614	0.636	0.30	0.36–0.43
Flinkkilä et al. [2]	5			0.23	0.18
Kreder et al. [13]	36			0.68	0.48

PACS picture archiving and communication systems, PDA personal digital assistant

Table 5.

Comparison of intra-observer agreement in the literature

	κ value
	Observers	Radius fracture	Ulnar styloid fracture	AO		Mayo
				Types	Groups
Bellinzona PACS	4	1	0.76	0.7	0.55	0.58
Bellinzona PDA	4	0.93	0.75	0.59	0.5	0.57
Jin et al. [29]	5			0.49 (0.45–0.57)	0.36 (0.34–0.41)
Ploegmakers et al. [27]	45				0.52
Olivera Filho et al. [12]	9				0.38 (with subgroups)
Andersen et al. [31]	4			0.58–0.70	0.341–0.45	0.30–0.63
Kreder et al. [13]	36			0.67–0.86	0.25–0.42 (with subgroups)

PACS picture archiving and communication systems, PDA personal digital assistant

This could imply that, when the initial treatment of a wrist fracture is based on a surgeon’s viewing of radiographs, there could be no difference between a large dedicated screen and a last generation phone display. Being the latter reliable, easily available and cheap, there are situations in which a quick and precise opinion can be of great benefit to patients, especially in all those countries without a dedicated telemedicine network between the local hospitals and the main trauma centres.

In our hospitals, the initial assessment of fractures is often done by casualty officers based on X-ray films taken by general practitioners in remote or poorly served areas. Inexperience and fear to underestimate injuries often leads to defensive medicine and unjustified transferrals. The use of MMS images could avoid long and expensive transferrals. The general practitioner can then exchange images with the specialist without the constraints of a sophisticated IT platform for the first and a fixed location for the second, providing to respect some basic rules to protect the patient personal details.

Moreover, it enables a radiological image sharing between specialists on PACS terminals without worrying of an excessively distorted image acquisition. All such advantages make the use of MMS images cost-effective [1, 3, 23]. The security of transmissions could be ensured by passwords [3] even if confidentiality is not an issue with the images taken directly with the phone camera, to avoid including the patients details.

Despite its progress, it is not clear whether telemedicine offers reliable diagnosis and allows planning treatment for bone trauma. Few studies using MMS for tele-consultation have looked specifically at orthopaedics, generally finding no significant difference in diagnostic accuracy between distance and face-to-face evaluation [1, 4, 20, 26, 40, 42, 43]. Most cases in these studies were soft tissues injuries [20, 26], which are fairly obvious to diagnose. Chandhanayingyong [9] investigated the accuracy of tele-consultation using MMS in orthopaedic patients with non- or minimally displaced fractures and demonstrated good reliability but poor diagnostic accuracy. There is no consensus regarding the acceptable level of diagnostic accuracy but it depends on the severity of the misdiagnosis. Our study highlighted the reasonable level of specificity (89%) and sensitivity (90%) of the mobile phone in evaluating a fracture to the radius despite the low sensitivity (77%) in detecting a fracture to the ulnar styloid, raising the concern for a type-2 error.

Nonetheless we believe that the less than ideal results published in this study as well as by Chandhanayingyong [9] should not penalise the potential of MMS consultation. In the latter case, he only studied patients with non- or minimally displaced fractures and this is a high-risk group for misdiagnosis even under ideal conditions. In complex wrist fractures the specialist may require further imaging, mostly CT scan [11]. In our study, the specialist consulted by MMS would not have considered the need for further investigations more than if he had seen the X-ray images on a PACS station. Therefore it can be argued that MMS images can be used not only for simple fractures but also for those which might require further imaging. We shall soon start a feasibility study also for mallet finger, metacarpal and proximal humeral fractures.

The addition of clinical information to X-ray images generally facilitates diagnosis but could also be misleading [1, 4, 9, 42] if an inexperienced doctor provides them. In our study we have carried out a solely radiographic analysis and intentionally excluded clinical information in order to avoid such possible source of interference. In the interpretation of our data it must also be considered that, given the low degree of inter-observer and intra-observer agreement for each of the distal radius fracture classifications, none can be used as only guide to treatment [2].

There are several limitations to our study. The first one is the possible recall bias caused by the second viewing, arbitrarily done 2 months after the first. Such effect is difficult to assess and can appear even after longer intervals [1]. Other limitations are the absence of a gold standard reference, such as a CT scan, to confirm or exclude a fracture and the modality of image capture with direct photograph of a PACS screen using the low-resolution camera of the telephone used for the on call service, which is 320 × 240 pixels. This was a deliberate choice based on the means and time available to our emergency service. It is known [23] how higher-resolution images produce higher satisfaction and, in fact, for complex CT and MRI images [9, 23] a minimum size of 640 × 480 pixels is required [23]. However higher-resolution images affect the transmission time [23] and have bigger file sizes. Kim [23] calculated that the transmission time for 320 × 240-pixel images was three times less than for higher resolution (1,144 × 880 pixels). Despite our device’s low resolution the inter-observer agreement did not differ and none of our observers judged the image quality as insufficient. Such is also the experience of Archbold [3], in whose study only one surgeon had difficulty in assessing images because of the low resolution. Image quality could be improved either using a phone device with a more powerful built-in camera or by the direct transferral of images from computer to phone in Joint Photographers Expert Group format [23]. However the latter would imply the transmission of the patient personal details as well as raising a matter about the interception of sensible information.