Monitoring the PACS Implementation Process in a Large University Hospital—Discrepancies Between Radiologists and Physicians

Abstract

Successfully introducing a new technology in a health-care setting is not a walk in the park. Many barriers need to be overcome, not only technical and financial but also human barriers. In this study, we focus on the human barriers to health-care information systems’ implementation. We monitored the acceptance of a Picture Archiving and Communication System (PACS) by radiologists and hospital physicians in a large Belgian university hospital. Hereto, questionnaires were taken pre-implementation (T1) and 1 year after the radiology department stopped printing film (T2). The framework we used to perform the study was the Unified Theory of Acceptance and Use of Technology. Main findings were that both groups were positive toward PACS prior to the introduction and that each group was even more positive at T2 with extensive PACS experience. In general, the ratings of the radiologists were higher than those of the physicians, as the radiologists experienced more of the benefits of PACS and had to use PACS throughout the day. Two factors were salient for predicting users’ intention to use PACS: the usefulness of PACS (performance expectancy) and the availability of support of any kind (facilitating conditions). The results show that our approach was successful. Both radiologists and physicians give evidence of an excellent level of user acceptance. We can conclude that the implementation of PACS into our hospital has succeeded.

Background

Introducing an information system (IS) in a health-care setting is not a straightforward story. Rather, it can be regarded as an adventure of which the final outcome can hardly be predicted, not even well after the initial deployment of the IS. A technology that is successfully implemented in one hospital may easily fail in another,¹ while a strategy which succeeded once may not be as successful the second time in another situation.²

To implement a health-care IS, some barriers need to be overcome. These arise from early on in the project, when the implementation is planned, until routine use of the IS has been established. According to Paré and Trudel,³ four distinct groups of barriers can be distinguished throughout the implementation process. These are portrayed in Table 1 along with some possible examples per type.

Table 1.

Barriers to IS Implementation in a Health-Care Organization

Barrier	Example
Project/economic	Funding issues
	Timeframe adherence
	Involvement of the end-user
	Choice of vendor
Technical	Operating System compliance
	Server and storage space
	Network capability
Organizational	Training issues
	End-user equipment availability
Behavioral/human	Acceptance and use of the IS by the end-user

In this article, we will focus on the human barriers to IS implementation, to be specific hospital physicians’ feelings toward the implementation of an IS. Earlier studies show that it is not just the IS that evokes negative feelings. Rather, it is the way the implementers or administrators deal with the users’ negative feelings towards the IS that can cause further resentment.¹ Initially, physicians can be positive towards the IS; however, this can change when they experience what the introduction truly encompasses. For example, they expect the system to be beneficial from day 1, forgetting that they will have to go through a learning process when they start to work with it. With the introduction of an IS, physicians might need to change their way of working, possibly altering the power relations between doctors and their nurses or clerks. Implementation of an IS can mean that more tasks have to be executed by the user himself and can no longer be delegated to supporting personnel, e.g., entering an order in a Computerized Physician Order Entry system or in an Electronic Patient Record, searching for radiological images on a Picture Archiving and Communication System (PACS)…. This may pose a problem, certainly when physicians’ use of the IS is mandatory, which could result in a feeling of threat towards losing autonomy, legitimacy, or status.⁴

In this work, the implementation of a Picture Archiving and Communication System is studied. With the advent of PACS, a range of new possibilities opened up both for radiologists and requesting physicians. In addition to the new and advanced image processing functions, we believe that the most important innovations of PACS are related to workflow improvements and the increased accessibility of radiological images. With PACS, images are available faster, images hardly ever get lost, and all authorized staff can access the images simultaneously,⁵ from within the hospital, home, and even the sunny beaches of Waikiki. In fact, the benefits of a PACS are tangible throughout all levels of the organization. Some of the benefits have been outlined in Table 2.

Table 2.

The Benefits of PACS on Different Levels Throughout the Organization

Level	Benefit
Management	Cost reduction6
Radiology department	Reduction of report turnaround time7
	Increased productivity3,8
	Higher job satisfaction4
	Lowered need for storage space
Physicians	Increased reliability of image delivery; significant reduction of the number of lost images and a faster availability of the images3,9
	Decreased time for image searching9
	Improved availability and accessibility of images5
Patients	Reduction in waiting time5
	Improved patient care3
	Reduction in average hospital stay10

Ultimately, the introduction of a PACS should lead to an improved level of service for the patients, who in our view should be the primary beneficiaries of any health-care information system.

The basic advantages of PACS lead to the expectation that PACS implementations are not likely to fail. This is confirmed in a follow-up study of Bauman and Gell.¹¹ They found that only 5.5% of the respondents had abandoned PACS or decreased its use. Now, with the technical progress of networks, workstations, computers… this number is only likely to decrease. However, not abandoning a PACS is not a valid criterion for success or failure. Only if an adequate amount of PACS potential is embraced by a critical mass of end-users can implementation be considered a success. To achieve this, some measures can be taken to improve PACS’ success. Johnson and Dye¹² described ten such measures; the most important for this study are:

• Not overselling PACS

• Addressing physical needs

• Identification of a project champion to lead the project

• The commitment of the upper management

We believe that it is also necessary to provide training so that the full potential of the PACS can be reached by each individual user.¹³ The method of training should be chosen very carefully¹⁴ as physicians are very busy and their learning to work with PACS will not be deemed a priority. Next to this, continuous support should be provided to the users, especially in the early days of PACS use.¹⁵ PACS implementers should also bear in mind that different users hold different views regarding PACS success.¹⁶

We consider PACS’ implementation success as PACS being accepted by the users. If an end-user uses PACS (almost) every day and has positive perceptions of PACS, it is safe to conclude that he or she accepts PACS. If this is true for a vast majority of the users, the implementation of PACS has succeeded. Users are more likely to accept PACS when they experience its benefits. To achieve the benefits of PACS, two conditions are paramount. First, the implementers need to provide training to the end-users so that they become proficient in PACS use. Second, the end-users need to integrate the use of PACS in their way of working. Some features of PACS can be tailored to fit in the existing way of working, but the end-users, the radiologists and the referring physicians, need to make the biggest efforts. They have to adapt their way of working to working with PACS, and for some, this also means learning to use a computer. Here, the importance of providing training to the end-users emerges. Users who are not proficient in working with the system will not be eager to adapt their way of working, as they experience no benefits of PACS use.

In IS literature, the acceptance of an IS by the users is operationalized as “use of the IS” or “Behavioral Intention (BI) to use the IS”. In the past, several technology acceptance models have been developed to explain and predict IS acceptance. A very powerful and parsimonious model is the Technology Acceptance Model (TAM).¹⁷ TAM has two predictors of technology acceptance: perceived usefulness and perceived ease of use of the technology, while other factors were left out of the model. For this study, we will use an elaboration of TAM, the Unified Theory of Acceptance and Use of Technology (UTAUT, Fig. 1) constructed by Venkatesh et al..¹⁸ UTAUT incorporates four predictors of “BI” or “use”: performance expectancy (PE, usefulness), effort expectancy (EE, ease of use), social influence (SI, pressure from peers/superiors), and facilitating conditions (FC, provision of support). Furthermore, four more variables were included which moderate the relationships between the predictors and BI or use: gender, age, experience with the technology, and setting (voluntary or mandatory use of the technology).

Fig 1.

The Unified Theory of Acceptance and Use of Technology.

This framework was used to monitor the acceptance of PACS in our university hospital.^19,20 The medical staff of the hospital (with a capacity of 1,169 beds) consists of approximately 600 physicians (of which 37 are radiologists) and 1,700 nurses. PACS has been introduced in different phases, starting with the radiology department. Less than 1 year after the introduction of PACS in the radiology department, the hospital went completely filmless.

While PACS was being planned in our hospital, some measures were taken to support the project. A multidisciplinary PACS project team was installed consisting of physicians, technicians, and engineers. The project team was set up to develop ownership of the oncoming PACS implementation project. The primary targets of the PACS project team were:

• To define PACS’ functional needs and to describe them in a tender

• To assess the different products proposed by the different vendors in regard to the defined functional needs

• To act as internal project management and thus become the sole reference within the radiology department concerning PACS-related issues both internally and towards external vendors

• To streamline the radiological workflow in optimizing links and interfaces between the utilized software components

• To develop support mechanisms for all end-users, both from within the radiology department and throughout the hospital for all physicians, e.g., an e-learning system

The PACS implementation process is outlined in Figure 2. As the PACS project team was the driving force behind the implementation of PACS, the radiological workflow was changed immediately with the introduction of PACS. In fact, the workflow changed in several steps to a digital way of working, first with the introduction of the Radiology Information System, then with the introduction of speech recognition, and then with the introduction of PACS. The physicians on the other hand could make the transition when they felt ready for PACS, anytime during the dual “analog film printing/digital PACS delivery” period. The radiologists were trained on working with PACS in a radiology-based expertise center under the supervision of application specialists. An e-learning system was developed for the physicians.¹⁴ This digital learning environment is accessible both from within and outside the PACS web viewer. However, two scenarios could have been a threat to the outcome of the PACS project: first, if the physician did not learn to work with PACS and, second, if the physician did not adapt his way of working. To anticipate these problems, we set up acceptance studies to monitor the acceptance and use of PACS throughout the hospital.^19,20

Fig 2.

Timeframe of the PACS project and timing of the questionnaires.

Methods

All potential PACS-using physicians were included in our study. The responses of radiologists were kept separate from the other physicians. The questionnaire was taken at two times: pre-implementation (T1) and about 1 year after the hospital went completely filmless (T2). A time frame of the study and the PACS project is depicted in Figure 2. The questionnaires were issued and collected through the internal mail system of the hospital.

The questionnaire consisted of different parts. The first part probed for demographic information (age, gender, specialty). The second part was the actual questionnaire, with scales and items of UTAUT¹⁸ for PE, EE, SI, FC, attitude toward use (ATT), self-efficacy (SE), anxiety (ANX), and BI. One extra scale appeared in both questionnaires: voluntariness of use (VOL) stemming from Innovation Diffusion Theory.²¹ The items were translated, and minor adaptations were made to fit it in our study. All items had to be assessed on a seven-point Likert scale, ranging from “complete disagreement (1)” to “complete agreement (7)”. At T2, one extra item questioning the self-reported frequency of use (USE) was added. This item had to be rated on a seven-point scale ranging from “never” to “daily.” The last part reserved some space for comments and wishes concerning PACS.

Results

At T1, 203 usable questionnaires (19 from radiologists) were returned and 159 at T2, of which 12 were from radiologists. Due to psychometric problems with the VOL scale, this scale was omitted from the analysis.

Figure 3 presents a graphical overview of the descriptive statistics. Data were analyzed using SPSS15©. The results of the analysis are displayed in Table 3.

Fig 3.

Graphical overview of the mean scale ratings per group and per time. USE was not measured at T1; for display reasons, the ANX scale was reverse coded so that “7” corresponds with “not at all anxious toward use of PACS”.

Table 3.

Significance Level of the Between-Groups Tests

Scale	Radiologists	Physicians	Radiologists vs. Physicians
	T1 vs. T2a	T1 vs. T2b	T1a	T2a
PE	p = 0.02	p < 0.001	p = 0.03	p = 0.002
EE	ns	p < 0.001	p = 0.001	p = 0.08c
SI	p = 0.07c	p < 0.001	p = 0.001	p = 0.002
FC	ns	p < 0.001	p < 0.001	p < 0.01
ATT	ns	p < 0.01	p < 0.001	p = 0.001
SE	ns	ns	p = 0.03	p < 0.01
ANX	p < 0.05	p < 0.001	ns	ns
BI	p < 0.05	p < 0.001	p = 0.001	p = 0.01
USE	N/Ad	N/Ad	N/Ad	p < 0.01

Only the significant values are reported

^aMann–Whitney U test

^bIndependent samples t test

^cMarginally significant

^dUse was not measured at T1

The analysis of the descriptive statistics revealed some interesting findings. First of all, we notice that in both groups, mean scale ratings improve over time. We see that the radiologists were already very positive toward the advent of PACS right from the beginning, and this only improved, leading to maximum scores on the BI and USE scales at T2. The improvement from T1 to T2 is even bigger in the physicians group. They were somehow less welcoming PACS; however, working with PACS made their perceptions of PACS much more positive. Neither of the two user groups felt anxious toward use of PACS. When comparing the radiologists and the physicians, we see that initially the radiologists scored higher on almost all scales, and the same picture arose at T2.

The analysis also revealed some differences between the male and female physicians. These are described in more detail in Duyck et al..¹⁹ However, most important here is that the male physicians rated PE higher at T1, while at T2 the female physicians experienced more pressure to use PACS and they were more anxious toward use of PACS.

To assess whether the implementation of PACS was successful, we calculated per scale the proportion of respondents that scored higher than neutral (“4”). We calculated proportions of all scales; these are displayed in Table 4. We estimate that SI, FC, and SE give no indication of acceptance of PACS by the end-user, however, we did include them because they give interesting information concerning other factors: SI on the amount of pressure to start using/use of PACS; FC on the work of the project team; and SE on the end-users’ estimated competency level of working with PACS.

Table 4.

Acceptance of PACS: Proportion of Respondents with a Score >4

Scale	Radiologists		Physicians
	T1	T2	T1	T2
PE	0.95	1.00	0.71	0.86
EE	0.79	0.92	0.58	0.84
SI	0.53	0.83	0.21	0.53
FC	1.00	1.00	0.54	0.93
ATT	0.95	1.00	0.76	0.88
SE	0.84	0.83	0.71	0.67
ANXa	0.89	1.00	0.89	0.98
BI	1.00	1.00	0.84	0.98
USEb	N/Ac	1.00	N/Ac	0.79

^aANX was reverse coded for clarity so that more than “4” corresponds with “not anxious”

^bFor use, the proportion displays the respondents with a score of ≥6 (this corresponds with the response “almost daily”)

^cUse was not measured at T1

The proportions give some more information on the results of the descriptive statistics. When we consider only the “acceptance scales” (PE, EE, ATT, ANX, BI, USE), we see that, on each scale and time, a vast majority (more than 70%) of the respondents give evidence of positive perceptions concerning PACS. There was only one exception: physicians at T1 on the EE scale. It was also important that the proportions increased (if there was still room) from T1 to T2. At T2, we notice that PACS is widely and very frequently used: 79% of the physicians indicated that they used PACS almost daily, while all radiologists indicated that they used PACS every day. The “non-acceptance scales” also show interesting information. It is obvious that the physicians could choose when to switch to PACS. Only 21% felt pressure to start using PACS at T1, and this proportion was still low at T2. The radiologists had no choice other than using PACS, which explains that more radiologists experienced social pressure to use PACS. On the FC scale, the work of the PACS project team is reflected: the radiologists were certain that they would get support if needed, as they were familiar with the members of the project team. There were fewer physicians convinced that they would get the necessary support at T1, but this figure almost doubled at T2. The proportion of radiologists who were convinced that they would be able to work with PACS was on both times slightly higher than of the physicians, and there were only minor changes from T1 to T2.

Then we performed a regression analysis to identify the key factors for PACS acceptance pre- and post-implementation and overall. Radiologists and physicians were analyzed as one group. The results are displayed in Table 5.

Table 5.

Regression Analysis with Behavioral Intention (Upper Part) and Use (Lower Part) as Dependent Variables

	T1 (n = 203)	T2 (n = 159)	Pooled (n = 362)
Dependent variable: behavioral intention
PE	0.41***	0.30***	0.36***
EE	0.14a	0.00	0.12*
SI	0.07	0.12a	0.12**
FC	0.18**	0.34***	0.26***
Adjusted R2	0.38	0.33	0.42
Model test	F(4,198) = 31.615, p < 0.001	F(4,154) = 20.137, p < 0.001	F(4,357) = 66.735, p < 0.001
Dependent variable: use
BI	N/A	0.12	N/A
FC	N/A	0.16a	N/A
Adjusted R2	N/A	0.05	N/A
Model test	N/A	F(2,156) = 4.874, p = 0.01	N/A

The values reported are standardized beta regression coefficients

*p ≤ 0.05; **p < 0.01; ***p < 0.001; ^ap ≤ 0.10

At both times and overall, PE was the best predictor of intention to use PACS, with FC as a good secondary predictor. EE and SI were only salient for predicting BI when the data were pooled over the two periods, but there were indications that EE on T1 and SI on T2 play a minor role in predicting the intention to use PACS. Neither BI nor FC were significant predictors of use of PACS. However, there was an indication that FC could play a minor role in predicting use of PACS, although the variance explained in use was very low (0.05). Variance explained in BI was acceptable.

Discussion

This study addresses the need expressed by van de Wetering et al.⁵ for another way of evaluating a PACS implementation. Here, end-users’ perceptions of PACS were assessed pre- and post-implementation to identify and anticipate possible causes for resistance against the implementation of PACS. Prior to the introduction of PACS, already in the planning phase, several measures were taken. A very important measure was the installation of a radiology-based PACS project team. This team was from the beginning (and still is) responsible for all aspects concerning PACS. The members of the project team gave PACS training to the radiologists. They also developed a digital learning environment¹⁴ which serves both as a training instrument for the physicians and as a support instrument for experienced and novice PACS users. The project team developed two more visual aids: a mouse pad and a blotting pad, each depicting summarized information on how PACS tools should be used. When PACS was announced and introduced in the hospital, the members of the project team visited the staff meetings of each service at three times. The reason for this was twofold: first, to introduce PACS and its possibilities and, second, to announce the support of the project team for the PACS project. We believe that the success of this labor-intensive approach is reflected in the results of our study.

The descriptive statistics show that both radiologists and physicians were receptive to PACS pre-implementation, and their ratings were even more positive at T2. The ratings of the radiologists on almost all scales were higher at both times. At T1, this can be attributed to the fact that the introduction of PACS would have an immediate impact on the job of the radiologists. The physicians could, albeit limited in time, choose when to switch to PACS; the radiologists had to switch to PACS immediately. Moreover, radiologists should have a better view on the possibilities of PACS than the physicians, as it is their domain. The higher ratings at T2 could be due to the fact that the radiologists use PACS more frequently and more thoroughly than other physicians. Radiologists have to work with PACS the whole day, while physicians only use PACS when they need to consult radiological images. The proportions displayed in Table 4 show that not only mean scale ratings improve from T1 to T2 (as shown in Fig. 3 and Table 3) but also that more of the end-users became more positive toward PACS.

As stated in other technology acceptance studies in medical settings,^22–24 the use of the technology depends more on its usefulness (PE) than on its ease of use (EE). It does not really matter how hard PACS is (to learn) to use, the physicians will employ PACS if it is useful for the job and patients. By choosing UTAUT, and not TAM, as a framework for monitoring the introduction of PACS, we gained additional insights. Our results stress the importance of providing support on the intention to use an IS. The availability of support is not only important when the technology is introduced; it becomes even more important when the users are already very experienced in using the technology. Our results also show that putting pressure on physicians to start using a new IS has no beneficial effect. However, when they are already using it, pressuring them to continue using the IS could surely be beneficial.

A limitation of this study is that we lack a measure of acceptance in the early stages right after the introduction of the IS. It could be that ease of use of a new technology and social influence to use a technology are especially salient when users start using a new technology. Here, the users had no hands-on experience with PACS, so they had to estimate PACS’ ease of use at T1. Pressuring physicians to try working with PACS might be either very beneficial or destructive for the acceptance of PACS. A measurement at about 1 to 3 months after the introduction might clarify these issues, leading to a better insight in the importance of these constructs.

Conclusion

A lot of effort was invested to make the implementation of PACS in our hospital a success. Next to preparing the infrastructure of the hospital, much energy was invested by the members of the PACS project team in making the end-users enthusiastic about PACS. This goal has been achieved. Both radiologists and physicians were positive toward the advent of PACS and even more positive with extensive PACS experience. Both groups had positive perceptions of PACS, with the radiologists being even more positive than the physicians. Two factors were extremely important for PACS to be accepted by the end-users: first, it was important that the usefulness of PACS was stressed during the implementation process, and second, the end-users had to sense that they could rely on support whenever problems should arise. The transition to PACS was completed within a year after go-live in the radiology department. The introduction of PACS into our hospital can be considered a success.