Abstract
A virtual medical imaging department is an innovative and demanding organizational model, to the extent that the underlying goal is to achieve a continuous and advanced organizational integration of human and physical resources, clinical data, and clienteles. To better understand the kind of benefits offered, we conducted a survey of three groups of users—radiologists, radiological technologists, and medical specialists—working in a five-site virtual organization. We received 127 valid questionnaires, for an overall response rate of 66%. The assessments vary according to the use made of the system. The scores for system quality and the quality of the data produced were markedly higher for intra-hospital use (respectively 7.9 and 8.7 out of 10) than for inter-hospital use (5.4 and 7.0). Despite the negative assessments they made of inter-hospital use, users maintained a positive attitude toward some type of virtual organization of medical imaging. Indeed, the score for Overall satisfaction with the system was very high, 8.9 out of 10. Moreover, the scores for Intended future use of the system were very high for both intra-hospital use (8.9) and inter-hospital use (8.7). We also found significant differences in perceptions among user groups.
Key words: PACS integration, PACS implementation, Integrating Healthcare Enterprise (IHE), Evaluation research
Introduction
Over the last few years, synergies have grown between the information technologies that digitize clinical data and provide access to telecommunication networks and the health care reforms that have tried to improve service access and continuity. This has spurred wider dissemination of computerized systems or picture archiving and communication system (PACS) in hospital medical imaging departments. Several studies have demonstrated that adoptions of this technology have become relatively smooth undertakings, despite the profound changes required to make the transition to digital data.1 The success of these implementations is due to the technical as well as organizational maturity attained by the technology. In terms of the technology, the hardware and software have proven value and have become very reliable. In terms of their place in the organization, system functionalities have achieved a good fit with the clinical business process requirements of medical imaging in hospitals. This guarantees that users will enjoy a range of benefits, something that has been confirmed in many recent studies. A positive impact is reported in terms of better X-ray image quality.2–6 PACS can thus improve the accuracy of clinical diagnoses, so it has a direct impact on quality of care.7 In more general terms, studies show how the new technology transforms work organization and clinical business processes, generating greater overall productivity.8–10 Higher productivity is observed among radiologists11–13 as well as radiological technologists.13,14 These productivity improvements stem from improved work processes,8,15 better data integration,2 a faster production of diagnosis reports,8,15,16 and shorter waiting times and hospital stays.3,15 Finally, economic studies have also been positive, since cost savings have been achieved.15,17–19
With this wide range of benefits, it is easy to understand why many stakeholders suggest wider use of this technology. One promising model is inter-hospital PACS, whereby an interoperable and integrated PACS serves several different hospital sites. Despite the distances between sites, system interoperability allows designers to create a multi-site network for exchanging clinical data in real time. Patient images can be dematerialized and viewed by radiologists in distant health facilities.20 The main impact of this organizational model has been the removal of the barriers in space and time that have traditionally structured professional work in medical imaging. In addition, since the system can integrate several sites, it can radically transform and reorganize how clinical data is managed, determining whether radiologists and technologists are needed on site and how patients gain access to continuity of care. Radiologists and technologists can intervene from a distant location, such that a remote radiologist can make an intervention with a patient who is then served in their local facility. This virtual and broader organization of medical imaging presents new opportunities for managing professional staffing shortages and reduced physical access to medical imaging services. This model is different from teleradiology, since it goes beyond occasional teleconsultations. A virtual medical imaging department is an innovative and more demanding organizational model, to the extent that the underlying goal is to achieve a continuous and more advanced organizational integration of human and physical resources, clinical data, and clienteles. Such a department should therefore provide greater benefits.
Yet this virtual model of medical organization is more complex and presents new and major challenges that may represent hurdles to the expected benefits. First, establishing a virtual organization is technically challenging, since it requires a telecommunications network that can continuously operate reliably and well across multiple and distant sites. In terms of the organization of clinical work, business processes and clinical practices need to be standardized across these sites if the clinical data and patients are to be exchanged efficiently. In this regard, a study was made of a virtual department of medical imaging in Australia that uses an integrated PACS/RIS Web application. The results clearly show the impact of these factors on user satisfaction.21
To better understand the kind of benefits afforded by this innovative model for the virtual organization of medical imaging, we conducted a survey of the users of an inter-hospital PACS system working in five health care institutions at eight different sites. These facilities served a population of 524,000 persons residing on a 21,554 km2 territory. At the time of the survey (2007), this well-established system had been up and running for 3 years. The survey sought to understand the perceptions of three groups of users—radiologists, radiological technologists, and medical specialists—of system benefits. In order to perform a thorough analysis of user perceptions, we used a recognized and tested analytical framework to examine a wide range of benefits associated with PACS.22,23 In addition, in order to analyze the specific nature of an inter-hospital PACS application, two types of uses were identified: intra-hospital use (meaning the traditional internal use of a system) and inter-hospital use (meaning use from at a distant site). The following section presents the conceptual model used to develop the survey questionnaire.
Conceptual Model
Since it was first developed in 1992, Delone and McLean’s model22,23 has been used in a wide variety of studies of information systems, including clinical data systems24 and, more specifically, PACS.21,25 This model has a wide variety of dimensions that can be used to make a methodical and thorough evaluation of all the benefits of an information system as perceived by users. The first set of benefits corresponds to the system’s intrinsic qualities: system quality, the quality of the data it generates, and the quality of available technical support. A second set of benefits concerns satisfaction with the system itself and with system use. This set is used to make assessments of general user satisfaction. A third set of benefits consists of those perceived by users at the individual and organizational levels. Figure 1 provides a list of the benefits assessed by this study as well as the assumed relationships between these variables. In the next section, we discuss how these benefits were measured and how their interrelationhips were tested.
Fig. 1.
Theoretical Research Model Adapted from DeLone & McLean (2003)23 and Paré et al. (2005)25.
Methods
Study Sites and Sample
A questionnaire survey was conducted in the first inter-hospital medical imaging network deployed in Quebec’s health care system. With a population of 7.5 million people, the province of Quebec has the second largest health care system in Canada. The five institutions using this inter-hospital PACS network operate at eight sites, have over 8,000 employees and count 208 medical specialists. Two of the institutions are medium-sized, specialized general hospitals located in urban settings. They have 433 and 215 beds, respectively. The other three institutions operate in remote areas far from urban centres, and each has less than 100 beds. In 2003, these institutions jointly purchased a PACS system and the system is still being operated jointly. The product, SYNAPSETM, is marketed by Fuji. At the time of this study, the Quebec’s health system intranet WAN was an ethernet-type LAN extension. Each hospital had a bandwith connection of 10 Mbs to the regionalized technocenter using a hierachical sytem management architecture for long-term storage. The technocenter had a 100 Mbs bandwidth capacity. Using packet prioritization and compression protocols, the informational infrastructure was designed to minimize the impact of large images traveling over low-bandwidth networks. A definable per modality compression format followed by decompression via the software at the workstation reduces transfer time and delays in an inter-hospital medical imaging network environment. At the time of purchase, the inter-hospital PACS system was seen as a tool that would remedy a serious shortage of radiologists and improve access to medical imaging services in outlying regions. Other objectives included improving the quality of medical imaging services and the continuity of care provided by partner facilities. Finally, it was hoped that the inter-hospital network would generate economies of scale through the acquisition and management of a single integrated system.
We conducted a mail survey of all users of the inter-hospital PACS. The targeted population included all radiologists (n = 15) and radiological technologists (n = 106) as well as all the medical specialists in seven different specializations (general surgery, orthopedics, internal medicine, nuclear medicine, pediatrics, respirology, and urology; n = 71) making intensive use of the medical imaging diagnostic service. In total, 192 persons were asked to participate. The survey was conducted between December 2006 and March 2007, 3 years after the end of the PACS implementation. The opinions received therefore came from seasoned users of a system that had reached cruising speed. The study was approved by the relevant university ethics authorities.
Defining the Perceived Benefits
The survey questionnaire (Appendix 1) was based on the DeLone and McLean model.22,23 First, the intrinsic qualities of PACS were measured in terms of the two contexts of use. Seven variables evaluated intra-hospital use: Ease of use (three items), Screen quality (three items), PACS-RIS integration (three items), Reliability (three items), Accessibility (two items), Perceived usefulness (three items), and Response time (two items). Four other variables measured inter-hospital use: Ease of use (two items), Inter-site integration (two items), Accessibility from remote sites (one item), and Response time (one item). The data quality generated by PACS was evaluated through four variables for intra-hospital use: Completeness (one item), Reliability and validity (two items), Availability (one item), and Safety (one item). The same four variables were also used for inter-hospital use, but we added a fifth variable to measure the Quality of inter-site integration of the data generated by the various centers (two items). Quality of technical support was assessed with one variable (five items) concerning the whole system, regardless of whether use was internal or external. As far as technical support was concerned, it was difficult to establish in which context the support was provided. As suggested by the DeLone and McLean model,22,23 we measured overall satisfaction (three items) without distinguishing type of use (intra-hospital or inter-hospital). Use was measured through a self-assessment made by each user. Intra-hospital use was measured with two variables: Frequency of use (one item) and Intensity of use (two items). Inter-hospital use was assessed with a variable measuring Frequency of use (one item). Benefits were measured in terms of improved Productivity (three items) and Quality of services (four items) for intra-hospital use and in terms of improved Productivity (one item) and Access (four items) for inter-hospital use. Finally, we added two variables to DeLone and McLean’s model22,23 that had been used in another study of the benefits of PACS:25 Confirmation of expectations of users (two items) and their intended future use of PACS (two items). All the answers were measured on 10-point Likert scales that ranged from “Strongly disagree” to “Strongly agree.”
Data Analysis
The internal consistency of each scale was validated using Cronbach’s alpha. Bivariate analyses were performed to confirm statistically significant differences between the three groups of users and the two contexts of use (intra-hospital and inter-hospital). T tests were performed using Mann–Whitney Kruskal–Wallis X2 tests for user type and Mann–Whitney X2 tests for type of use. Finally, two analyses of multiple linear regressions tested the relationships hypothesized in our theoretical model. An initial regression model tested the extent to which specific variables could explain overall user satisfaction; these variables measured the intrinsic qualities of PACS, the quality of the data generated, the quality of technical support, confirmation of expectations, and user type. A second regression model tested the degree to which variables for overall satisfaction, benefits, and confirmation of expectations could explain users’ intention to use the system in the future.
Results
Response Rate and Respondent Profile
We received 127 valid questionnaires, for an overall response rate of 66%. The rate varied by type of user: we received valid questionnaires from 80% (12/15) of the radiologists, 72% (76/106) of the radiological technologists, and 55% (39/71) of the medical specialists. Table 1 provides the distributions of these three groups by age, sex, and institution.
Table 1.
Socio-Demographic Characteristics of Respondents
| Radiologists (n = 12) % | Specialists (n = 39) % | Technologists (n = 76) % | |
|---|---|---|---|
| Age (years) | |||
| 20–30 | 0 | 8 | 33 |
| 31–40 | 46 | 30 | 23 |
| 41–50 | 8 | 35 | 25 |
| 51 and over | 46 | 27 | 19 |
| Sex | |||
| Female | 45 | 19 | 84 |
| Male | 55 | 81 | 16 |
| Hospital | |||
| Hospital A | 50 | 48 | 44 |
| Hospital B | 33 | 40 | 35 |
| Three other hospitals | 17 | 12 | 21 |
Psychometric Qualities of the Variables
The internal consistency coefficients (Cronbach’s alpha) of the measuring scales proved highly satisfactory. As shown in Table 2, they vary between 0.73 and 0.97, surpassing the desired minimum threshold of 0.70.26 One measure, Ease of use in an inter-hospital use context, fell below this threshold, with a coefficient of 0.64. We nevertheless maintained this variable for our analyses since it approaches the minimum threshold. Also, according to the literature, ease of use and perceived usefulness are the most significant variables explaining new technology adoption.27,28
Table 2.
Descriptive Psychometric and Statistical Qualities of the Scales Used to Measure Assessments of PACS by Type of Use
| Construct variables | Intra-hospital use (n = 125) | Inter-hospital use (n = 30) | Mann–Whitney X2 (p value) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| # Items | Alpha | Mean | SD | # Items | Alpha | Mean | SD | ||
| System quality | |||||||||
| Ease of use | 3 | 0.92 | 8.50 | 1.48 | 2 | 0.64 | 6.47 | 2.41 | 21.37 (0.000) |
| Screen monitor quality | 3 | 0.84 | 7.97 | 1.71 | n/a | n/a | n/a | n/a | n/a |
| Integration | 3 | 0.91 | 7.97 | 2.06 | 2 | 0.95 | 5.27 | 2.65 | 23.22 (0.000) |
| Reliability | 3 | 0.86 | 7.50 | 1.93 | n/a | n/a | n/a | n/a | n/a |
| Accessibility | 2 | 0.82 | 7.32 | 2.28 | 1 | – | 4.58 | 3.78 | 8.64 (0.003) |
| Perceived usefulness | 3 | 0.89 | 8.18 | 1.80 | n/a | n/a | n/a | n/a | n/a |
| Response time | 2 | 0.77 | 7.48 | 1.96 | 1 | – | 4.80 | 2.20 | 30.32 (0.000) |
| Overall measure | 19 | 0.92 | 7.89 | 1.36 | 6 | 0.76 | 5.41 | 2.03 | 36.67 (0.000) |
| Data quality | |||||||||
| Completeness | 1 | – | 8.98 | 1.48 | 1 | – | 7.80 | 2.23 | 10.09 (0.001) |
| Reliability and validity | 2 | 0.87 | 8.77 | 1.47 | 2 | 0.97 | 7.40 | 2.26 | 11.71 (0.001) |
| Availability | 1 | – | 8.68 | 1.86 | 1 | – | 6.50 | 2.76 | 19.66 (0.000) |
| Safety | 1 | – | 8.34 | 1.87 | 1 | – | 7.23 | 2.31 | 6.81 (0.009) |
| Integration of information | n/a | n/a | n/a | n/a | 2 | 0.95 | 6.17 | 2.92 | n/a |
| Overall measure | 5 | 0.92 | 8.70 | 1.45 | 7 | 0.91 | 6.97 | 2.21 | 20.01 (0.000) |
| Quality of technical support | 5 | 0.97 | 8.55 | 1.70 | |||||
| Use | |||||||||
| Frequency | 1 | – | 9.45 | 1.66 | 1 | – | 7.03 | 2.62 | 40.22 (0.000) |
| Intensity | 2 | 0.74 | 8.42 | 1.93 | |||||
| Overall satisfaction | 3 | 0.95 | 8.87 | 1.70 | |||||
| Benefits | |||||||||
| Productivity | 3 | 0.87 | 8.40 | 2.09 | 1 | – | 4.78 | 2.41 | 39.40 (0.000) |
| Quality | 4 | 0.86 | 7.77 | 1.82 | |||||
| Access to care | n/a | n/a | n/a | n/a | 4 | 0.79 | 5.65 | 2.09 | n/a |
| Overall measure | 7 | 0.91 | 7.90 | 1.91 | 5 | 0.55 | 5.46 | 1.66 | 39.99 (0.000) |
| Confirmation expectations | 2 | 0.78 | 7.97 | 1.85 | 1 | – | 5.66 | 2.58 | 20.25 (0.000) |
| Intended future use | 2 | 0.73 | 8.92 | 1.73 | 1 | – | 8.66 | 2.57 | 0.323 (0.570) |
PACS Assessments by Type of Intra-hospital and Inter-hospital Use
Respondents’ assessments of PACS varied by type of use. As shown in Table 2, their overall assessment of the system’s qualities was clearly higher for intra-hospital use (7.9) than for inter-hospital use (5.5; p = 0.0000). The highest scores of system quality were in the variables Perceived usefulness and Ease of use for intra-hospital use (8.2 and 8.5, respectively), while the lowest scores were for Accessibility and Response time in inter-hospital use (4.6 and 4.8, respectively)—even falling below the mid-point of our 10-point scale. The overall measure of data quality was also higher for intra-hospital use (8.7) than inter-hospital use (7.0; p = 0.000). Two variables for inter-hospital use gave the lowest scores: Integration of information (6.2) and Availability of information (6.5). In contrast, all variables measuring data quality in intra-hospital yielded scores above 8/10.
Differences in assessments for intra-hospital and inter-hospital use were also found in system use, both in terms of level of satisfaction and number of users. Scores on both of the measures used to estimate intra-hospital use were very high (Frequency of use, 9.5 and Intensity of use, 8.4), while Frequency of use in inter-hospital applications was lower, at 7.0 (p = .000). On the other hand, while intra-hospital use of PACS was widely reported by the survey’s respondents, only 24% (30/127) indicated that they used the system to call up digital images from a site other than their own. Only six radiologists (50%), 16 medical specialists (42%), and eight medical imaging technologists (11%) used PACS for this purpose. The low percentage of technologists can be explained by the fact that the organization of their work continues to be traditional, meaning that they mainly work in activities that take place in their own hospital.
The differences between these two contexts of use were also apparent in our analysis of perceived benefits. Intra-hospital use received positive assessments in terms of Increased productivity (8.4) and Quality of care (7.8), while inter-hospital use received poorer assessments. In the latter instance, Increased productivity scored a low 4.8 (p = 0.000) out of 10. The score for Access to services was also low (5.7), underscoring PACS’ incapacity to fully support a practice occurring at several sites in a virtual network. This poor assessment of perceived benefits echoes the negative assessments reported above of system quality and the data produced in a context of inter-hospital use. This observation was confirmed in the gap between assessments of Confirmation of expectations in intra-hospital use (8.0) and inter-hospital use (5.7; p = 0.000). Finally, despite the differences in the assessments made in the two contexts, users still reported that they believe in PACS, inasmuch as high scores were recorded for Intended future use in both intra-hospital (8.9) and inter-hospital (8.7) contexts.
PACS Assessments by User Group
Table 3 shows the perceptions of PACS reported by the three user groups in terms of work in an intra-hospital context. Even though statistically significant differences between the three groups were not found in the overall measure of system quality, significant differences were observed in five of the variables of the overall measure (Ease of use, Screen quality, Integration, Perceived usefulness, and Response time). With one exception (Screen quality), technologists reported the highest level of satisfaction, followed by radiologists and then medical specialists. The same order was found in scores for the overall measure of data quality (p = 0.006). Similarly, satisfaction with the Quality of technical support was high among technologists (9.1) but markedly lower for radiologists (7.5) and medical specialists (7.9; p = 0.000). The other dimension of the model that provided statistically significant differences was Use, in terms of both Frequency and Intensity. No differences were found in the model’s other variables. Overall satisfaction levels and assessments of system benefits were therefore similar for all three user groups.
Table 3.
Analysis of Differences of Opinion of PACS User Groups in an Intra-hospital Context
| Radiologists (n = 12) | Specialists (n = 38) | Technologists (n = 75) | Mann–Whitney | ||||
|---|---|---|---|---|---|---|---|
| Kruskal–Wallis | |||||||
| Construct variables | Mean | SD | Mean | SD | Mean | SD | Chi-squared (Asymp. sig) |
| System quality | |||||||
| Ease of use | 8.22 | 2.14 | 7.96 | 1.61 | 8.82 | 1.20 | 9.76 (p = 0.008) |
| Screen monitor quality | 8.44 | 2.15 | 7.33 | 1.81 | 8.22 | 1.51 | 10.31 (p = 0.006) |
| Integration | 7.97 | 2.26 | 6.77 | 2.27 | 8.51 | 1.71 | 13.00 (p = 0.002) |
| Reliability | 7.44 | 1.50 | 7.61 | 1.89 | 7.45 | 2.03 | 0.465 (p = 0.792) |
| Accessibility | 8.38 | 1.87 | 7.14 | 2.05 | 7.23 | 2.42 | 4.54 (p = 0.103) |
| Perceived usefulness | 8.35 | 2.08 | 7.16 | 2.23 | 8.67 | 1.25 | 14.46 (p = 0.001) |
| Response time | 7.67 | 1.64 | 6.85 | 2.12 | 7.76 | 1.87 | 6.34 (p = 0.042) |
| Overall measure | 8.08 | 1.79 | 7.34 | 1.46 | 8.13 | 1.16 | 0.892 (p = 0.640) |
| Data quality | |||||||
| Completeness | 8.42 | 2.43 | 8.74 | 1.48 | 9.21 | 1.25 | 4.99 (p = 0.082) |
| Reliability and validity | 8.38 | 2.39 | 8.54 | 1.56 | 8.96 | 1.21 | 2.00 (p = 0.369) |
| Availability | 8.33 | 2.39 | 8.32 | 2.36 | 8.93 | 1.39 | 1.65 (p = 0.439) |
| Safety | 8.30 | 2.63 | 8.34 | 1.82 | 8.34 | 1.81 | 0.152 (p = 0.927) |
| Overall measure | 8.38 | 2.39 | 8.49 | 1.50 | 8.86 | 1.15 | 10.32 (p = 0.006) |
| Quality of technical support | 7.45 | 2.57 | 7.88 | 1.62 | 9.07 | 1.36 | 21.99 (p = 0.000) |
| Use | |||||||
| Frequency | 9.33 | 2.31 | 9.16 | 1.90 | 9.62 | 1.39 | 4.74 (p = 0.09) |
| Intensity | 8.75 | 1.97 | 7.66 | 2.01 | 8.42 | 1.93 | 11.93 (p = 0.00) |
| Overall satisfaction | 8.56 | 2.35 | 8.35 | 2.26 | 9.19 | 1.07 | 3.80 (p = 0.15) |
| Benefits | |||||||
| Productivity | 8.69 | 2.52 | 8.59 | 2.18 | 8.26 | 1.98 | 3.67 (p = 0.16) |
| Quality | 7.62 | 2.27 | 7.13 | 2.28 | 8.11 | 1.37 | 3.98 (p = 0.14) |
| Overall measure | 8.07 | 2.25 | 7.77 | 2.06 | 8.18 | 1.49 | 0.802 (p = 0.670) |
| Confirmation of expectations | 8.21 | 2.49 | 7.44 | 2.28 | 8.19 | 1.42 | 3.19 (p = 0.20) |
| Intended future use | 8.29 | 2.55 | 8.76 | 2.16 | 9.10 | 1.25 | 1.08 (p = 0.58) |
Table 4 shows user groups’ perceptions in inter-hospital use. In this case, due to the limited number of respondents we were unable to test for statistically significant differences among the groups, but the results are nevertheless revealing. First, the differences of opinion between user groups are more pronounced than in the previous table. Overall, radiologists held the most negative opinions, followed by medical specialists and technologists. The lowest scores on system quality came from the radiologists, particularly with respect to system integration and data integration. Indeed, the integration of system components provided the lowest scores on our 10-point scale: PACS/RIS integration (2.8) and dictation integration (1.7). Scores for Data integration (one of the dimensions of data quality) were very low for the data (4.0) as well as the imaging (3.2). Similarly, perceived benefits in terms of Increased productivity received very poor scores from the radiologists (3.5) but also from the other two groups. The radiologists’ score for Confirmation of expectations is still disappointing (3.7), but the corresponding score from the other two groups was 6.0. In contrast, scores for Intended future use of the system were high for all three groups: radiologists (8.0), medical specialists (8.6), and technologists (9.3).
Table 4.
Analysis of Differences of Opinion Between PACS User Groups in an Inter-hospital Context
| Radiologists (n = 6) | Specialists (n = 16) | Technologists (n = 8) | ||||
|---|---|---|---|---|---|---|
| Construct variables | Mean | SD | Mean | SD | Mean | SD |
| System quality | ||||||
| Ease of use | ||||||
| Image reception | 5.20 | 3.56 | 6.75 | 1.92 | 7.63 | 2.45 |
| Image transmission | 4.60 | 2.01 | 6.23 | 3.22 | 7.17 | 3.19 |
| Integration | ||||||
| PACS/SIR systems | 2.83 | 2.14 | 5.75 | 2.26 | 7.71 | 1.11 |
| PACS/SIR/dictation systems | 1.67 | 1.21 | 5.46 | 2.26 | 7.00 | 2.28 |
| Accessibility from home | 5.20 | 3.63 | 3.27 | 3.54 | 8.75 | 0.96 |
| Response time | 3.50 | 1.05 | 5.00 | 2.39 | 5.38 | 2.26 |
| Overall measure | 3.64 | 1.24 | 5.37 | 1.99 | 6.83 | 1.57 |
| Data quality | ||||||
| Completeness | 6.50 | 3.62 | 7.88 | 1.82 | 8.63 | 1.41 |
| Reliability and validity | ||||||
| Validity | 6.50 | 3.62 | 7.63 | 1.78 | 8.63 | 1.41 |
| Well organized and presented | 6.50 | 3.62 | 6.75 | 2.24 | 8.38 | 1.41 |
| Availability | 5.00 | 3.35 | 6.56 | 2.76 | 7.50 | 2.07 |
| Safety | 6.50 | 3.62 | 7.19 | 2.23 | 7.88 | 1.13 |
| Integration | ||||||
| Data/patient | 4.00 | 4.24 | 6.25 | 2.86 | 7.71 | 1.50 |
| Images/patient | 3.17 | 2.40 | 6.27 | 2.66 | 8.13 | 1.25 |
| Overall Measure | 5.38 | 3.03 | 6.98 | 1.97 | 8.16 | 1.28 |
| Use | ||||||
| Frequency | 7.67 | 2.66 | 7.63 | 2.42 | 5.38 | 2.56 |
| Benefits | ||||||
| Productivity | 3.50 | 2.88 | 5.47 | 2.10 | 4.33 | 2.42 |
| Access | 5.71 | 2.48 | 5.32 | 2.09 | 6.28 | 1.90 |
| Overall Measure | 5.27 | 1.59 | 5.31 | 1.87 | 5.91 | 1.32 |
| Confirmation of expectations | 3.67 | 2.42 | 6.06 | 2.35 | 6.43 | 2.70 |
| Intended future use | 8.00 | 3.52 | 8.63 | 2.75 | 9.29 | 0.76 |
Multivariate Analyses
The interrelationships postulated by DeLone and McLean’s22,23 model were tested with regression analyses. Both regression models performed well, explaining 71% and 48%, respectively, of the variation observed in Overall satisfaction and Intended future use. In the first model, the variables System quality, Data quality, and Confirmation of expectations explained much of the variance in Overall satisfaction. The Medical specialists variable was also statistically significant, indicating differences of opinion between this user group and the other two groups. The second regression model revealed that Intended future use of the system can be explained by two variables: Overall satisfaction and Perceived benefits (Table 5).
Table 5.
Linear Regressions of Overall Satisfaction and Intended Future Use of PACS
| Dependent variables | Independent variables | Estimate parameter | SD | Standardized coefficients | t value | p value |
|---|---|---|---|---|---|---|
| Overall satisfactiona | Constant’s parameter | 0.526 | 0.075 | 6.967 | 0.000 | |
| Ease of use | 0.094 | 0.060 | 0.135 | 1.572 | 0.119 | |
| Screen monitor quality | 0.005 | 0.063 | 0.950 | |||
| Integration | −0.073 | 0.048 | −0.129 | −1.527 | 0.130 | |
| Reliability | 0.016 | 0.063 | 0.024 | 0.252 | 0.802 | |
| Accessibility | −0.034 | 0.056 | −0.057 | −0.614 | 0.541 | |
| Perceived usefulness | 0.075 | 0.059 | 0.116 | 1.273 | 0.206 | |
| Response time | 0.154 | 0.050 | 0.227 | 3.086 | 0.003 | |
| Data quality | 0.253 | 0.069 | 0.348 | 3.680 | 0.000 | |
| Quality of technical support | 0.008 | 0.055 | 0.012 | 0.137 | 0.891 | |
| Confirmation of expectations | 0.207 | 0.055 | 0.309 | 3.765 | 0.000 | |
| Radiologists | 0.167 | 0.095 | 0.109 | 1.762 | 0.082 | |
| Medical specialists | 0.168 | 0.074 | 0.156 | 2.273 | 0.025 | |
| Intended future useb | Constant’s parameter | 0.288 | 0.117 | 2.455 | 0.016 | |
| Overall satisfaction | 0.567 | 0.099 | 0.535 | 5.695 | 0.000 | |
| Overall benefits | 0.154 | 0.074 | 0.177 | 2.079 | 0.040 | |
| Confirmation of expectations | 0.059 | 0.067 | 0.087 | 0.879 | 0.318 | |
| Radiologists | 0.106 | 0.112 | 0.066 | 0.943 | 0.348 | |
| Specialists | −0.086 | 0.074 | −0.082 | −1.166 | 0.246 |
aModel’s overall significance: F = 21.24, p = 0.000, R2 = 0.743, R2 adjusted = 0.708
bModel’s overall significance: F = 22.37, p = 0.000, R2 = 0.500, R2 adjusted = 0.477
Discussion
The goal of this study was to analyze the range of benefits provided through the virtual organization of medical imaging, which brings together resources from several health care institutions in a single PACS system. This new organizational form is achieved by deploying an interoperable and integrated PACS that allows several remote sites to function as a single entity, sharing equipment, clinical teams, and data and providing care to a common clientele. We conducted a survey that collected data on the perceptions of three user groups familiar with such a system: radiologists, radiology technologists, and medical specialists. The process yielded five main findings.
First, the assessments made of PACS vary according to the use made of the system. The scores for system quality and the quality of the data produced were markedly higher for intra-hospital use than for inter-hospital use. To our knowledge, no other study has measured levels of satisfaction for both types of use in a given population, but our results support the research in this field, insofar as the studies that have examined user satisfaction with intra-hospital use also found very high rates of satisfaction.4–6 Our findings are similar to those from a study conducted in an inter-hospital context that also reported a high level of dissatisfaction with response time.21 This problem has since been resolved through upgrades to the telecommunication network’s capacity. Second, this difference in assessments is also clearly seen in system use (only one respondent in four reported inter-hospital use). Our results from the satisfaction measure reveal that clinicians remain a demanding user group when it comes to the usefulness and ease of use of a technology. Scores for the variables Perceived usefulness and Ease of use were much higher for intra-hospital use than they were for inter-hospital use. This finding agrees with an earlier study of information technology adoption that identified these two factors as the main reasons explaining the adoption of new technologies.27,28 These two factors were even more critical in this case, since our sample of radiologists were under no obligation to analyze examinations made at the other sites. At the time of our survey, remote reading was still optional. Given the shortage of radiologists practicing in outlying regions, it was difficult to make this type of activity mandatory, even more so since the quality of the inter-hospital PACS was considered poor. This situation should improve with the deployment of a regional on-call medical imaging system. Third, it is important to stress that despite the negative assessments they made of inter-hospital use, users maintained a positive attitude toward some type of virtual organization of medical imaging. Indeed, the score for Overall satisfaction with the system was very high: 8.9 out of 10. Moreover, the scores for Intended future use of the system were very high for both intra-hospital use (8.9) and inter-hospital use (8.7). This is a particularly significant finding, given that the variable Confirmation of expectations was very high for intra-hospital use (8.0) but much lower for inter-hospital use (5.7). Fourth, we found significant differences in perceptions among user groups. These findings strongly resemble those reported by Paré et al.,25 who found greater satisfaction among radiologists and technologists than among medical specialists. However, our findings are different from those reported by Tually,21 who observed a higher level of satisfaction among radiologists than technologists, particularly in terms of system quality, data quality, service quality, and overall satisfaction. Fifth, our regression analyses confirmed that, as suggested by the DeLone and McLean model,22,23 there are indeed strong relationships between system quality, user satisfaction, and intended future use.
Before concluding, it is worth mentioning the limits of this study. This was first and foremost a cross-sectional analysis of data collected at a specific moment in time. In this context, our regression analyses measured associations and links between variables, but they cannot demonstrate true causality. In addition, our findings came from the analysis of a single virtual medical imaging organization which represented a first trial of this form of organization in Quebec’s health care system. It is important that this type of study be repeated in order to validate our findings and better understand this phenomenon.
In conclusion, we would like to reiterate the overall positive attitude expressed by the three main groups using hospital medical imaging. They were seasoned users who had the opportunity to fully test this new form of organization: inter-hospital PACS. Despite a certain amount of dissatisfaction that may be due to the innovative nature of this emerging form of virtual organization, we nevertheless found sustained user participation in the project, and the level of intended future use was high. This is encouraging news for the deployment and sustainability of this new form of virtual organization of medical imaging practices.
Acknowledgments
The authors would like to thank all of the physicians and technologists who took the time to complete our survey and make this study possible. As well, we are indebted to the Canadian Institutes of Health Research for the financial support provided to this research project.
APPENDIX
Table 6.
Survey Questionnaire
| Intrinsic qualities of PACS | |
| Ease of use | Generally speaking, the PACS system is easy to use. |
| It is easy to master PACS system functionalities. | |
| The graphical interfaces of PACS are clear and easy to understand. | |
| It is easy to find images from the other facilities in PACS.a | |
| It is easy to send images and information to other facilities using PACS.a | |
| Screen quality | The PC screens available in our facility are of acceptable quality. |
| The specialized PACS screens are of acceptable quality. | |
| The quality of the screens encourage me to use PACS. | |
| PACS-RIS integration | The PACS/RIS subsystems are well integrated. |
| The PACS/RIS/Dictation subsystems are well integrated. | |
| Joint use of PACS/RIS/Dictation makes the work easier. | |
| When the data on a patient comes from different facilities, the PACS/RIS systems provide well-integrated information.a | |
| When the data on a patient comes from different facilities, the PACS/RIS/Dictation systems provide well-integrated information.a | |
| Reliability | PACS is rarely offline because of technology breakdowns. |
| Unexpected PACS service outages rarely occur. | |
| PACS use is uninterrupted because the system is bug-free. | |
| Accessibility | We have a sufficient number of PACS work stations. |
| I have rarely had to wait for access to a work station in order to consult PACS. | |
| I have easy access to PACS from my home computer.a | |
| Response time | I have the impression that images download quickly. |
| We have quick access to images from other facilities.a | |
| With the exception of image management, PACS responds quickly. | |
| Perceived usefulness | Overall, PACS provides a complete range of functionalities that support my work as a professional. |
| My clinical practices are very well harmonized with PACS. | |
| Using PACS is compatible with all aspects of my tasks. | |
| Data quality | |
| The PACS images produced locally in your facility (or externally) are: | |
| Complete | |
| Reliable and precise | |
| Well organized and carefully presented | |
| Available in a timely manner | |
| Secure and confidential | |
| Quality of technical support | |
| The staff in your hospital who provide technical assistance for PACS | |
| Are easy to reach | |
| Provide quick service | |
| Are competent | |
| Pay attention to user needs | |
| Are able to find satisfactory solutions | |
| Use | |
| I often use the PACS system to fulfill my functions. | |
| I use the PACS system for the great majority of my clientele. | |
| I use a wide range of the PACS system’s functionalities. | |
| I often use the PACS system to obtain clinical data from other facilities. | |
| Overall satisfaction | |
| Overall, my experience using the PACS system has been satisfactory. | |
| I enjoy using the PACS system in my work. | |
| Overall, using PACS is more satisfying than using the old system. | |
| Benefits | |
| Productivity | My personal productivity has improved since I have been using the PACS system. |
| Using PACS allows me to save time. | |
| Using PACS has saved me some of the time I used to spend moving about. | |
| The work load has increased due to the greater number of requests for medical imaging examinations, particularly those originating outside my facility.a | |
| Quality of services | When compared to the use of photographic films, our use of PACS has improved quality of care. |
| Use of PACS has improved the quality of medical imaging diagnoses. | |
| PACS has reduced the time between examination requests and the delivery of results. | |
| PACS use has led to improved relations between professionals. | |
| Inter-hospital access | PACS has made medical imaging services more accessible.a |
| Regional PACS has reduced waiting times for patients in my region.a | |
| The PACS system has reduced patient transfers between facilities.a | |
| PACS has reduced the impact of staffing shortages in medical imaging.a | |
| Confirmation of expectations | |
| My personal experience with PACS is better than I had expected. | |
| Generally speaking, the benefits of PACS are in line with my initial expectations. | |
| The ease of access to images from other facilities is better than what I had initially hoped.a | |
| Intended future use | |
| I want to continue using the PACS system in my clinical activities. | |
| I want to continue using PACS to obtain imaging data from other facilities.a | |
| If I could, I would like to become more proficient at using the PACS system. | |
aItems measuring the inter-hospital context
References
- 1.Reiner BI, Seigel EL, Sidiqqui K. Evolution of the digital revolution: a radiologist perspective. J Digit Imaging. 2003;16(4):324–330. doi: 10.1007/s10278-003-1743-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Wetering R, Batenburg R, Versendaal J, Lederman R, Firth L. A balanced evaluation perspective: picture archiving and communication system impacts on hospital workflow. J Digit Imaging. 2006;19(Suppl. 1):10–17. doi: 10.1007/s10278-006-0628-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Crowe B, Sim L. Implementation of a radiology information system/picture archiving and communication system and an image transfer system at a large public teaching hospital–assessment of success of adoption by clinicians. J Telemed Telecare. 2004;10(Suppl 1):25–27. doi: 10.1258/1357633042614339. [DOI] [PubMed] [Google Scholar]
- 4.Yu P, Hilton P. Work practice changes caused by the introduction of a picture archiving and communication system. J Telemed Telecare. 2005;11(Suppl 2):104–107. doi: 10.1258/135763305775124867. [DOI] [PubMed] [Google Scholar]
- 5.Pilling JR. Picture archiving and communication systems: the users’ view. The British Journal of Radiology. 2003;76:519–524. doi: 10.1259/bjr/67551353. [DOI] [PubMed] [Google Scholar]
- 6.Watkins J. A hospital-wide picture archiving and communication system (PACS): the views of users and providers of the radiology service at Hammersmith Hospital. Eur J Radiol. 1999;32:106–112. doi: 10.1016/S0720-048X(99)00134-5. [DOI] [PubMed] [Google Scholar]
- 7.Reiner BI, Seigel EL, Hooper FJ. Accuracy of interpretation of CT scans: comparing PACS monitor displays and hard-copy images. Am J Roentgenol. 2002;179:1407–1410. doi: 10.2214/ajr.179.6.1791407. [DOI] [PubMed] [Google Scholar]
- 8.Lepanto L, Paré G, Aubry D, Robillard P, Lesage J. Impact of PACS on dictation turnaround time and productivity. J Digit Imaging. 2006;19(1):92–97. doi: 10.1007/s10278-005-9245-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Fleishon HB, Bhargavan M, Meghea C. Radiologists’ reading times using PACS and using films: one practice’s experience. Acad Radiol. 2006;13:453–460. doi: 10.1016/j.acra.2005.12.014. [DOI] [PubMed] [Google Scholar]
- 10.Reiner BI, Seigel EL. PACS in radiology—technologists’ productivity when using PACS: comparison of film-based versus filmless radiography. Am J Roentgenology. 2002;179(1):33–37. doi: 10.2214/ajr.179.1.1790033. [DOI] [PubMed] [Google Scholar]
- 11.Bryan S, Weatherburn GC, Watkins JR, Buxton MJ. The benefits of hospital-wide picture archiving and communication systems: a survey of clinical users or radiology services. Br J Radiol. 1999;72:469–478. doi: 10.1259/bjr.72.857.10505012. [DOI] [PubMed] [Google Scholar]
- 12.Kato H, Kubota G, Kojima K, Hayashi N, Nishihara E, Kura H, Aizawa M. Preliminary time-flow study—comparison of interpretation times between PACS workstations and films. Comput Med Imaging and Graph. 1995;19(3):261–265. doi: 10.1016/0895-6111(95)00010-N. [DOI] [PubMed] [Google Scholar]
- 13.Reiner B, Seigel E, Scanlon M. Changes in technologist productivity with implementation of an enterprisewide PACS. J Digit Imaging. 2002;15(1):22–26. doi: 10.1007/s10278-002-0999-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Reiner BI, Seigel EL, Carrino JA, Goldburgh MM. SCAR radiologic technologist survey: analysis of the impact of digital technologies on productivity. J Digit Imaging. 2002;15(3):132–140. doi: 10.1007/s10278-002-0021-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Nitrosi A, Borasi G, Nicoli F, Modigliani G, Botti A, Bertolini M, Notari P. A filmless radiology department in a full digital regional hospital: quantitative evaluation of the increased quality and efficiency. J Digit Imaging. 2007;20(2):140–148. doi: 10.1007/s10278-007-9006-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Mariani C, Tronchi A, Oncini L, Pirani O, Murri R. Analysis of the X-ray work flow in two diagnostic imaging departments with and without a RIS/PACS system. J Digit Imaging. 2006;19(Suppl 1):18–28. doi: 10.1007/s10278-006-0858-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fang YC, Yang MC, Hsueh YS. Financial assessment of a picture archiving and communication system implemented all at once. J Digit Imaging. 2006;19(Suppl 1):44–51. doi: 10.1007/s10278-006-0632-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Reddy AS, Loh S, Kane RA. Budget variance analysis of a departmentwide implementation of a PACS at a major academic medical center. J Digit Imaging. 2006;19(Suppl 1):66–71. doi: 10.1007/s10278-006-0852-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Charvet-Protat S, Thoral F. Economic evaluation and organizational benefits of picture archiving and communication systems. J Radiol. 1998;79(12):1453–1459. [PubMed] [Google Scholar]
- 20.Aas IHM. Organizational cooperation in teleradiology. J Telemed Telecare. 2005;11(1):45–50. doi: 10.1258/1357633053430449. [DOI] [PubMed] [Google Scholar]
- 21.Tually P, Stavrianou C, Walker J. User acceptance of the Web-based distribution of radiology services in regional and remote centres of Western Australia. J Telemed Telecare. 2005;11(Suppl 2):93–95. doi: 10.1258/135763305775124948. [DOI] [PubMed] [Google Scholar]
- 22.DeLone WH, McLean ER. Information systems success: the quest for the dependant variable. Inform Sys Res. 1992;3:60–95. doi: 10.1287/isre.3.1.60. [DOI] [Google Scholar]
- 23.DeLone WH, McLean ER. The DeLone and McLean Model of Information Systems success: a ten year update. J Manage Inf Syst. 2003;19(4):9–30. [Google Scholar]
- 24.Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc. 2003;10(3):235–243. doi: 10.1197/jamia.M1094. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Paré G, Lepanto L, Aubry D, Sicotte C. Toward a multidimensional assessment of picture archiving and communication system success. Int J Technol Assess Health Care. 2005;21(4):471–479. doi: 10.1017/S0266462305050658. [DOI] [PubMed] [Google Scholar]
- 26.Nunally J. Psychometric Methods (2nd Edition) New York: McGraw Hill; 1978. [Google Scholar]
- 27.Schepers J, Wetzels M. A meta-analysis of the technology acceptance model: investigating subjective norm and moderation effects. Inf Manage. 2007;44(1):90–103. doi: 10.1016/j.im.2006.10.007. [DOI] [Google Scholar]
- 28.Barki H, Paré G, Sicotte C. Linking IT implementation and acceptance via the construct of psychological ownership of information technology. J Inf Technol. 2008;23(4):269–280. doi: 10.1057/jit.2008.12. [DOI] [Google Scholar]