High-Volume Teleradiology Service: Focus on Radiologist Satisfaction

Abstract

In 1998 we surveyed our radiologists on teleradiology satisfaction. Results were generally positive. In 2002 we experienced a sevenfold case increase in teleradiology volume. The present study surveyed the radiologists again. The hypothesis was that, with increased case volume and radiologist experience with the system, ratings would increase. Image quality was excellent/good, although plain film and ultra sound (US) had more fair/poor ratings. Monitors, navigation, image processing, and Web-based reporting were rated as excellent/good. The voice-recognition system was rated poorly. Diagnostic confidence was about the same as for film. Exceptions were magnetic resonance imaging (MRI) US, and plain film. Up to 10% of cases are unreadable because of poor image quality, not enough images, or inadequate patient history. Overall, the radiologists are satisfied, although some improvements can be made.

TELERADIOLOGY is probably the most mature and well-established clinical specialty within telemedicine¹ and generally, diagnostic accuracy has been found to be equivalent to traditional film-based display.^2,3,4 In many ways, it is difficult to separate teleradiology from PACS (picture archiving and communications systems) in terms of the research findings and clinical practice, and thus many of the findings that pertain to PACS also pertain to teleradiology.⁵ However, there are some aspects of teleradiology that do make it different from PACS, and thus there are different parameters that need to be assessed. For example, with PACS there is a significant focus on workflow, the technologist, and time and costs saved.^6,7 Workflow and savings are certainly relevant to teleradiology, but the impact on the technologist (at least at the consulting site) is probably less of an issue. The impact on the radiologists who must read the teleradiology cases can, however, be significant. In many respects, from the perspective of the radiologist, PACS is simply a shift from one display medium to another, but studies have shown that radiologist workflow improves as well, because there are fewer interruptions, an increased sharing of work, faster/better access to old images and reports, and less paperwork.⁸ Although the increase in workflow may leave time for radiologists to read more images, unless the hospital itself has an increase in the need for imaging services, the actual volume of cases is not likely to change significantly simply as a function of going filmless. Teleradiology, however, typically represents an increase in case reading volume and thus generally requires some adjustments to be made in department operations and management. How much of a change is required depends, of course, on the type of teleradiology services provided. Primary interpretations and on-call coverage are likely to have more of an impact than the occasional second opinion or quality assurance cases.

Workstations, for PACS and/or teleradiology, have been improving steadily over the years, but they are not yet optimal.⁹ For example, compared to the traditional light box and film display, monitors are not as bright, have different MTF (modulation transfer function) properties, and have lower contrast ratios, more noise, less space to present images at full resolution, and different tone scales (ie, gamma). Assessing diagnostic accuracy and what influences it, however, are not the only factors to consider when assessing the impact of teleradiology or PACS. One important measure for assessing the impact on the radiologist of providing teleradiology (and PACS) services is to measure user satisfaction.^4,10,11,12 In the long-run, even if diagnostic accuracy is the same as with the traditional viewing medium, if the user is not satisfied with the system as whole (ie, the display, the interface, the reporting system, etc.) productivity is likely to suffer, fatigue is likely to increase, and usage may actually decrease rather than increase.¹³

We have been performing teleradiology at our institution since 1996. An initial feasibility study¹⁴ connected our Department of Radiology to a Community Hospital about 100 miles away. A brief survey of the radiologists interpreting the cases found that, in general, they were satisfied with the number of images transmitted per case, image quality, the timeliness of the consultation, and their level of diagnostic confidence. Overall, the radiologists thought that 95% of the teleradiology consults were successful, and based on these positive results we embarked on the development of a telemedicine network that included a very large teleradiology component.¹⁵ From 1996 to the end of 2001 there was an approximate doubling of cases (see Fig 1). During those early years cases (n = 9609) were being received from only 5 rural sites. In 2002, 13 additional teleradiology sites were incrementally added to the network, resulting in a sevenfold increase in the number of cases received. From January 2003 through April 2003 over 20,000 cases have been received, and now nearly 5,000 are being diagnosed each month.

Figure 1.

Total number of teleradiology cases received each year from 1996 through April of 2003.

Teleradiology is covered 24 hours a day, seven days a week, relying on staff radiologists during regular hours and senior residents for late-night and weekend coverage. In 1998 we conducted a survey of the radiologists doing teleradiology to gauge their satisfaction with the operation.¹⁶ The goal of the present study was to survey our radiology staff in 2002 after teleradiology case volume had increased significantly and more individuals had extensive experience with teleradiology and soft copy reading in general. The overall hypothesis was that with the increase in volume and soft copy reading experience, subjective ratings would increase because the radiologists were using the teleradiology operation more, thus gaining increased familiarity and comfort.

MATERIALS AND METHODS

The present survey included many of the same questions as the previous one,¹⁶ although some were revised to better reflect the current situation (ie, more modalities are read now) and some new questions were added. The questions addressed reading volume and modalities, image quality, diagnostic confidence, interface issues, and key positive and negative aspects of the teleradiology program. The survey was developed in-house and tested for validity and reliability before it was sent out. Surveys were distributed to all staff (23 full-time attendings) and the 10 third- and fourth-year residents in the Radiology Department (first- and second-year residents do not read teleradiology cases). Responses were anonymous. When questions were the same as in the 1998 survey, the present results were compared to the earlier results for statistically significant differences (chi-squared analyses). The teleradiology system we use is RadWorks (GE Medical Systems, Waukesha, WI). The voice-recognition software we use for reporting is MedSpeak (IBM Corp., Armonk, NY). We also use a Web-based reporting system, developed in-house, that takes the entered diagnostic reports and transmits them automatically to the referring site.

RESULTS

A total of 31 (94%) surveys were returned; 21 respondents (15 attendings [65%], 6 residents [60%]) indicated that they had read teleradiology cases. Those who do not read teleradiology cases did not complete the rest of the survey, so the results are based on the data from the 21 respondents who do read teleradiology cases. For questions included on both the 1998 and 2002 surveys the data from the attendings and residents were compared for statistically significant differences. When there were no statistically significant differences the data were combined and results were reported based on all data. Where differences were observed the results are presented for both groups of readers.

Most respondents to the survey (67%) started reading cases in 2001 or later. Fifty-three percent read more than 60 assigned cases per day, with attendings reading significantly more than residents (χ² = 11.44, df = 5, P = 0.0434). On the 1998 survey only 6% of respondents read more than 60 cases per day (see Fig 2), with the difference in reported cases read increasing significantly from 1998 to 2002 (χ² = 57.86, df = 5, P < 0.0001). To date, 19% have read over 1,000 cases total, 34% 600–1,000 cases, 38% 200–600, and 9% fewer than 200 total. Throughout the history of this program, digitized plain film computed radiography (CR) (59%) and computed tomography (CT) (28%) have made up the bulk of cases received from the referring sites. Ultrasound (US) (10%), mammography (2%), magnetic resonance imaging (MRI) (0.5%), and nuclear medicine (0.5%) account for the remainder. The type of exam read by most of the respondents is CT (n = 20), followed by digitized plain film (n = 15), US (n = 14), and CR (n = 13) (see Fig 3). Telemammography cases are read by three individuals, MRI by two, and nuclear medicine by one (more than one modality can be read by an individual). In 1998 there were significantly fewer modalities read than in 2002, with CT the most common, followed by plain film and US, and no other modalities read (χ² = 14.7, df = 6, P < 0.05).

Figure 2.

Percentage of respondents indicating the number of teleradiology cases read per day from the 1998 survey (solid bars) and the 2002 survey (hatched bars).

Figure 3.

Number of respondents in 1998 and 2002 reading different modalities via teleradiology.

For each modality, image quality was generally rated as excellent or good (Table 1), although digitized plain film (29%), digitized ultrasound (39%), and digitized nuclear medicine (33%) have more fair and poor ratings than the other modalities (χ² = 124.49, df = 6, P < 0.0001). Some respondents often indicated that CR images were of superior quality when compared to the hardcopy CR images read in house. Some of the referring sites have newer CR units than our Radiology Department. Direct Digital Imaging and Communication in Medicine (DICOM) ultrasound images were also noted in the comment section to have better image quality than the digitized ones (digitized US represent the majority received). In 1998 the survey did not break out image quality by modality because only CT, plain film, and US were read. In 1998 63% rated image quality for these modalities as excellent or good. Combining the 2002 ratings for CT, plain film, and US gives 73% excellent or good ratings. The 10% increase was not statistically significant (z = 0.77, P > 0.05).

Table 1.

Image Quality Ratings for Each Imaging Modality

	CT (n = 20)	US (n = 18)	Film (n = 17)	CR (n = 15)	Mammo (n = 3)	Nuc Med (n = 3)	MRI (n = 1)
Excellent	25%	11%	6%	33%	67%	0%	0%
Good	60%	50%	65%	67%	33%	67%	100%
Fair	15%	22%	18%	0%	0%	33%	0%
Poor	0%	17%	11%	0%	0%	0%	0%

Note: In parentheses, n = the number of respondents rating the modality. Film and most of the US cases are digitized.

System friendliness/usability was also rated for a variety of interface parameters. The monitors (3 megapixel flat-panel) were rated as excellent/good by 86% of the respondents; system navigation as excellent/good by 48%; image processing excellent/good by 50%; Web-based reporting interface excellent/good by 70%; and voice-recognition system for reporting excellent/good by 10%. The 1998 survey only asked about system friendliness/usability in a general way without breaking it out in specific system components, so the two surveys cannot be compared.

Diagnostic confidence for reading with the various modalities compared to traditional film reading was for the most part rated as about the same (Table 2). The exceptions are digitized US and digitized plain film, which are associated with a greater number of lower confidence ratings. The only modality receiving any ratings of better than clinical (non-teleradiology) reading was CR (13%). The correlations between diagnostic confidence and image quality ratings were moderate. Because MRI, mammography, and nuclear medicine had 3 or fewer readers, we did not calculate the correlation coefficients, but for CT the correlation was 0.414, for US it was 0.567, for plain film it was 0.589, and for CR it was 0.520. A correlation of 0.50 is generally regarded as a moderately strong relationship between two variables. Diagnostic confidence did not correlate with whether the respondent was a resident or an attending (less than 0.5 in all cases).

Table 2.

Diagnostic Confidence Ratings for Each Modality Compared to Traditional Film Reading.

	CT (n = 20)	US (n = 18)	Film (n = 17)	CR (n = 15)	Mammo (n = 3)	Nuc Med (n = 3)	MRI (n = 1)
Much better	0%	0%	0%	0%	0%	0%	0%
Better	0%	0%	0%	13%	0%	0%	0%
Same	70%	35%	41%	67%	100%	100%	0%
Lower	30%	59%	47%	20%	0%	0%	100%
Much lower	0%	6%	12%	0%	0%	0%	0%

Note: Film and most of the US cases are digitized.

The 1998 survey did not break out diagnostic confidence by modality, but overall the results were similar to 2002. Confidence was rated as about the same as film by 42% of the respondents, and lower than film reading by 68% (for CT, plain film, and US only). Combining the “about the same as clinical” for CT, plain film, and US and combining the lower than ratings for the same three modalities for 2002 yields 50% rating the same as clinical reading and 50% lower than clinical reading, yielding no statistically significant difference in diagnostic confidence for these modalities from 1998 to 2002 (χ² = 1.3, df = 1, P > 0.05).

Sixty-two percent of the respondents felt that there had been cases that they were not able to read via teleradiology, but for 94% of those the number of unreadable cases has been less than 10%. In 1998 26% said that there were cases they were unable to read, so there was a significant increase in the number of people reporting unreadable cases from 1998 to 2002 (z = 2.06, P < 0.05). Image quality was cited by 92% of those who could not read some images as the reason, 85% cited not having enough images, and 46% cited not enough clinical history (more than one reason could be marked). Interestingly, both residents and attendings cited image quality, but only attendings cited not enough images or not enough clinical history. In 1998 100% cited image quality, 20% not enough images, and 20% not enough clinical history as reasons for not being able to read cases (more than one reason could be marked). The difference in distribution of reasons between 1998 and 2002 was statistically significant (χ² = 32.61, df = 2, P < 0.0001).

The final two questions on the survey asked whether the radiologists had been given any information about the rural hospitals being served and whether the radiologists would be interested in understanding the situation of those rural hospitals. Only 33% of the respondents answered yes to the former question and 71% yes to the latter question. There was no statistical difference between residents and attendings about whether they had been given information about the rural sites, but the residents (17%) were significantly less likely (χ² = 12.34, df = 1, P < 0.001) than the attendings (93%) to be interested in understanding the situation of the rural hospitals.

The final question was an open-ended request to list the key positive and negative points of the teleradiology program and offer any suggestions for improvement. Seventeen people provided responses. One of the main positives noted was the fact that teleradiology represents a significant source of revenue for the department (n = 7). Another positive (n = 5) is that it represents an improvement in health care and a service to rural Arizona (ie, community service). On the clinical side it was noted that the diversity of cases is quite high and thus represents a good source of material for training the residents (n = 4). A frequent comment on the negative side was the lack of feedback or follow-up on the cases after the report has been sent to the referring site (n = 4). Also on the negative side was the need for certain aspects of the operation to be made more efficient (especially the voice-recognition system) and thus less time-consuming (n = 9). Three of the six residents indicated that they thought teleradiology was placing too much extra work on them, especially during the night, although two of the attendings noted that there should be more resident involvement. Other suggestions followed along the same lines with requests for more personnel to help cover the service (n = 3) and obtain another workstation to allow more people to participate (n = 6). Four people noted the need to train the technologists at the rural sites with respect to obtaining better quality images (image quality and patient positioning).

DISCUSSION

Between 1998 and 2002 our teleradiology program experienced a significant increase in the volume of cases referred for interpretation and an increase in the number of modalities included in the referrals. Although we were not able to compare all survey parameters between 1998 and 2002, we observed very few changes in terms of opinions regarding the teleradiology operation from a technical standpoint, contrary to our initial hypothesis that overall satisfaction would increase with increased experience. Diagnostic confidence remained about the same, with most people rating confidence about the same as traditional film reading. The exceptions were with those modalities where digitization of the original films was required (ultrasound and plain film) and image quality was rated lower, as was diagnostic confidence. This is not surprising, and corroborates a number of studies that have found that image quality affects diagnostic confidence and accuracy in radiology and other clinical specialties.^17,18 In fact, Taplin et al¹⁸ found that poor image quality (especially with respect to patient positioning) was strongly associated with interval breast cancers when cases of ductal carcinoma in situ were included. In our survey the main reason for not being able to read a case and provide a diagnosis for the referring clinician with an interpretation was poor image quality. Although the percentage of unreadable cases is relatively low (<10% for those saying there were unreadable cases), there is certainly room for improvement. As some of the radiologists noted in the comment section, this could easily be addressed by better training of the technologists at the rural sites. In some cases, however, this is difficult because the lack of qualified clinical personnel, and high turnover rates in the rural areas are often cited as one of the main reasons for developing telemedicine and teleradiology programs.¹⁹ One potential remedy, of course, is to use the Arizona Telemedicine Network for medical education in addition to clinical services.²⁰

One aspect that has changed since the inception of our teleradiology program was the need for greater participation on the part of more radiologists. Until the beginning of 2002 the majority of teleradiology cases were after hours. With the dramatic increase in volume at the beginning of 2002 it was necessary to add two full-time staff into the weekly schedule and to assign specific radiologists to read teleradiology cases each day. Greater involvement by the residents may prove to be a very useful teaching mechanism given the variety of cases seen and the greater exposure of soft copy reading in general. However, most of the residents felt that they were already spending too much time reading the teleradiology cases (especially at night) and did not necessarily view this as an educational opportunity. The lack of specific mechanisms for receiving feedback from the rural clinicians about the cases may also represent a potential barrier for using teleradiology as an educational tool. Although the types of cases received often have more diverse pathologies than what is seen on-site, the lack of feedback regarding the correctness of the diagnosis or whether the diagnosis had an effect on patient treatment, there is little in the way of learning opportunities with respect to diagnostic accuracy. Again, this is something easily addressed but not necessarily easy to carry out. Because our teleradiology uses a Web-based reporting system, it would be easy enough for the rural clinicians to simply e-mail feedback about some of the cases (especially if such a request is made by the teleradiologist). However, time becomes the limiting factor. Many rural clinicians are simply too overworked to take the time to provide feedback on telemedicine cases.

Overall, we found very few differences in the various ratings between the attendings and the residents other than the number of teleradiology cases being read. However, the final two questions did reveal something very interesting. Although only 33% of the residents had received any background on the rural sites (the same for attendings), the majority (83%) of the residents did not want to get more information to understand the rural situation. In contrast, most of the attendings (93%) did want to get more information to understand the rural situation. In some respects the feeling that most of the residents viewed teleradiology as a burden or as taking up too much of their time especially during the night likely contributed to their response to this question, whereas the attendings (who read significantly more cases) tended to view teleradiology as a good opportunity for the department in terms of being a revenue source and a good source of teaching material. They therefore had a more positive response to the question.

CONCLUSIONS

Teleradiology is clearly a beneficial service to rural sites, and in general our radiologists are satisfied with the program. There are some technical aspects that could be improved, and eliminating the need to digitize films would improve image quality and reduce the number of unreadable cases.