A Balanced Evaluation Perspective: Picture Archiving and Communication System Impacts on Hospital Workflow

Abstract

Around the world, hospitals are faced with both budget and regulatory pressures, forcing them to re-examine the way clinical practice is carried out. Proposed technologies that provide workflow enhancements include Picture Archiving and Communications Systems (PACS); however, is PACS really effective in improving hospital workflow and the flow onto patient care, and how should this be evaluated? An acknowledged and successful approach for organizational evaluation is the Balanced Scorecard (BSC), providing the fundamental features for assessing organizations from various perspectives. In this research, the impact of PACS on the workflow of a large public hospital in Melbourne, Australia, is examined using an adapted version of the BSC. Empirically, this model was applied as an evaluation instrument through a series of in-depth interviews with PACS users. Results show that PACS did improve hospital workflow considerably and that the organizational alignment of PACS in hospitals is an important critical success factor.

Background

Hospitals are moving toward filmless operations commonly investing in Picture Archiving and Communication System (PACS).¹ PACS goes beyond the operational boundaries of radiology by supporting different departments to access medical images and reports and hence to optimize patient care;² thus, it is considered as not just a repository of data, but also a “workflow-integrated imaging system that is designed to streamline operations throughout the entire patient care delivery process.”³ Furthermore, effective use of PACS can shorten the time for diagnosis, improve the efficiency and quality of the overall health care delivery process, and make the workflow as simple as possible.^3–6 However, although it is known that PACS makes images available at an earlier time, hard evidence to support the connection between earlier receipt of images, clinical decision making, and clinical action through PACS images is lacking.^7,8

There is an emerging need for evaluating PACS implementations because health care providers are facing both growing demand for improved care and higher expectations of service delivery.⁹ Until now, the diversity in PACS implementations leads to difficulties in applying standard methods for the evaluation of PACS. Some studies provide insights concerning the value of PACS in terms of return on investment and other kinds of quantitative measurements.^2,7,10,11 It is questionable however, if these measures are most adequate for measuring payoffs yielded by an enterprise-wide PACS implementation because it neglects the intangible value created for stakeholders.^2,12 Although it is recognized that general technological and organizational advances have been achieved over the last decades with the implementation of PACS, there is less consensus about the level of impact on patient care. Studies predominately focus on the quantitative, analytical aspects, whereas most of the potential gains are of a qualitative nature and thus hard to quantify. In addition, evaluations tended to focus on specific parts of the workflow implications of PACS implementations with a death of holistic evaluations.

This article is based on a case study at a major public hospital in Melbourne, Australia, which recently implemented a hospital-wide PACS to improve their workflow and patient care. The hospital is a major teaching, research, and tertiary referral center situated in Melbourne's central business district and provides a wide variety of community services. The hospital includes around 450 beds, 42 wards, and 16 operating theaters, and employs ∼3,000 employees across 19 discrete campuses. It performs roughly 65,000 imaging exams (from all modalities) annually. In the early 1990s, the hospital commented a PACS implementation in several departments, and it recently expanded the system hospital-wide. This implementation was never evaluated, however, despite the impression that it has a positive impact on patients, staff, and a broad range of processes throughout the hospital. In 2004, the initiative for evaluation was taken, providing an opportunity to investigate the impacts of a PACS on the workflow of the hospital and to apply a holistic approach to an evaluation of the PACS system.

Methods

A successful approach to an integral evaluation of organizational performance is the Balanced Scorecard (BSC) as developed 15 years ago by Kaplan and Norton.^13,14 The BSC is a performance management model that provides executives with a comprehensive framework that translates a company's strategic objectives into a coherent set of measures. This provides a comprehensive view of the most relevant issues of the business. In its original design, the BSC includes performance measures from four interrelated perspectives: (1) financial, (2) internal business process, (3) customer, and (4) learning and growth. The BSC is concerned with evaluating performance both with regard to return to shareholders and relationships with customers. The learning and growth perspective provides the impetus to sustain the ability to change and improve the other perspectives. The four different perspectives oblige senior managers to jointly reflect on operational measures. Hence, the BSC also indicates whether improvement in one area may have been achieved at the expense of another.

Much interest in the BSC currently exits within the health care industry. As the industry experience with the BSC grows and successes are shared, the use of the BSC in health care is expanding.^14,15 Although the BSC has been applied for organizational evaluations in health care,¹⁵ it has never been applied in the evaluation of PACS in hospitals. The implementation of the BSC allows organizations to adapt the different perspectives that are relevant to how they measure success based on the organization's strategy and vision. Consequently, the scorecard creates performance measurements tailored to the needs of organizations and links them with strategic plans. The BSC is therefore adaptive by design and thus should be applicable to the evaluation of a broad range of organizational components, including PACS implementations. Because strategy is central to the BSC, it is important to consider hospital strategy before applying it to PACS. It has been suggested that for hospitals, a focus on the nonmarket-driven components of the corporate strategy formulation process is most important. Therefore, the aspirations and values of executives in charge of the organization and the organization's obligation to society are central.¹⁶ Following this, in this research, the BSC is adapted to incorporate a wide range of nonmarket performance measures according to the hospital's strategic perspectives. Second, to evaluate a PACS implementation, the BSC requires an adaptation that incorporates components that are essential to the workflow of a patient's clinical journey. Therefore, in this research, the perspectives of the BSC were transformed based on their relevance to workflow, PACS, and consistent with hospital strategies. This leads to the following translation of the original BSC perspectives:

1. Clinical business process, as a translation of the internal business process perspective.

2. Patient, as a translation of the customer perspective.

3. Quality and transparency, as a translation of the financial perspective.

4. Information systems, as a translation of the learning and growth perspective.

The first two perspectives are similar to the original BSC, but the Quality and Transparency and the Information System perspectives are radically transformed compared with the original financial and learning and growth perspective, respectively. To fit the PACS evaluation, they now capture considerable changes in simplicity and transparency of workflow, the extent to which one can monitor and check the status of patients in the clinical process, and the importance of the system as an enabler of the workflow. If the four perspectives are connected as in the BSC, a new framework is obtained: the PACS-BSC model (see Fig. 1). Although the adaptation process was theory driven, the majority of claimed and frequently mentioned benefits by large PACS vendors concerning workflow impacts can be mapped on each of these four perspectives. This justifies the adjustments made to the original BSC. Because hospital goals of clinical excellence and community service are outcomes that are difficult to quantify, as are the intangible value created by PACS to patients, radiologist, and clinicians in terms of quantitative measures, we decided to elaborate the outlined perspectives as qualitative “themes” rather than quantitative performance “measures.” Instead of defining standard performance measures, our evaluation of PACS will be guided through the use of “principal themes” that originate from literature. Hence, this leads to a qualitative method for describing and analyzing the outcomes associated with most elements of our PACS-BSC model. Despite these modifications, the PACS-BSC model remains in accordance with Norton and Kaplan's original intentions of the BSC in that the four perspectives complement each other and all are related to the central vision and strategy. These four perspectives and their relationship are shown in Figure 1.

Fig 1.

The PACS-BSC model.

Data were collected through a series of in-depth, semistructured interviews with PACS users and other medical staff at a large public hospital in Melbourne, Australia. The research was conducted under the approval of the Department of Information Systems Human Ethics Advisory Group, University of Melbourne. Interviewees were selected using a systematic, nonprobabilistic technique to maximize insights from different respondents who cover each PACS-BSC model perspective. This resulted in a variety of responses that contributed valuable insights concerning the impact of PACS on specific aspects of the hospital workflow. A total of 11 interviews where held with staff members, project management, and executives from different departments including medical imaging, emergency department, the cardiothoracic care center, and physiotherapy. The interviews were recorded using a digital audio recorder with consent. These recorded interviews were then transcribed and provided to the interviewees for confirmation purposes. To analyze these data thoroughly and to use them to validate the PACS-BSC model, qualitative coding was used.¹⁷ The data were reviewed on three different occasions using open, axial, and selective coding, respectively. Open coding is a first step in providing a high-level overview of the data. The second pass (axial coding) is mainly about critically reviewing the initial codes and searching for missing elements. These new codes are created subsequently and other codes are adjusted when new insights arise during this process. The final phase of the coding process is selective coding that consists of systematic linking the different categories. The three coding techniques were used for the analysis process in an iterative way: New codes and themes appeared after new interviews and new insights were gained through each method of coding.

Results

The coding process performed on the 11 interview records resulted in the following selective codes, categorized according to the four perspectives of the PACS-BSC model:

• Clinical business processes: diagnosis process, time-savings and image-based clinical action, organizational communication, and examination request and report turnaround time.

• Quality and transparency: simplicity and transparency, quality of workflow, and agile workflow.

• Information system: availability and accessibility, PACS integration, and system robustness.

• Patient: patient waiting time and patient throughput and flow.

These codes are discussed in more detail below, including excerpts from the interviews to clarify the particular views of respondents.

From all interviews, it was confirmed that with the implementation of PACS, the diagnosis process has changed considerably in terms of efficiency and value for work. Images are all in digital, browser-based format. Radiologists analyze the images on their computer screen rather than by hanging films on a viewing box. The following excerpt from a radiologist clarifies this view:

I can display them how I want to display them and adjust windows and levels and in some respects it is actually much easier for me to stack the images, just one on top of the other and singing through them.

All authorized staff can access images simultaneously once they are stored into PACS, which has been beneficial for the staff. Previously, only radiologists had access to the images and provided reports to other hospital workers who did not necessarily see the original image. Because of the hospital-wide PACS, all staff can now access the images simultaneously, allowing collective decision-making concerning proper treatment of patients. However, although the ability to make a shared diagnosis has improved, no conclusive evidence was reported concerning considerable time savings for clinicians and earlier initiation of image-based clinical action. Some respondents, however, stated that PACS made the treatment of patients easier, because they do not have to wait for radiology to produce images and do not have to go down to the library to search for specific film bags.

Interviewees explained that, in some cases, PACS made the treatment of patients quicker. However, the total clinical treatment remains highly dependent on a lot of other organizational factors too, such as doctor availability, busyness of the department, availability of operating rooms, seriousness of the complaint, patient flows, and whether an official report of the radiologist is required. PACS does lead to less verbal organizational communication between the departments and a reduced need for communication. All information is stored into the system and the status concerning data is available in real time for all authorized staff. Significant clinical cases that are brought up either by the radiologist or the clinicians are still communicated face to face. Doctors from the emergency department can directly discuss a patient with the colleague who will be looking after the image up on the ward. Also, clinicoradiological meetings that take place in the hospital are performed in a more efficient manner. In accordance with the mentioned efficiencies above, several steps that make up the overall report turnaround time have been improved. The examination requesting process is still the same, although more incoming requests for imaging were noticed. After the images have been acquired and stored into PACS, they can be accessed by the radiologist who also has immediate access to the request that is scanned into the system.

Workflow at the hospital has become simpler and transparent as a result of PACS implementation. Nearly all steps that were associated with the hard copies and film bags have been eliminated. Images are stored into a central image database and made accessible to all the departments through a Web server.

...it removes a lot of the steps in the workflow, all of the film-handling steps are taken out of the workflow and that makes the workflow simpler...

Although less physical steps are involved in the examination of patients in radiology, closely monitoring these steps in terms of quality has become more important. Thus, it has become essential to check whether images have actually arrived from the PACS system to the right patient map/directory. PACS allows status checking of patients and examinations from any computer in the hospital along the workflow. This was seen as a major help in terms of patient and time management. Clinicians and nurses can easily check whether the image has been taken, stored, and reported by the radiologist. Also, PACS changed work scheduling in some areas of the hospital. Radiologists schedule their reporting session more efficiently now that they do not have to worry about hanging any of the films and the film bag.

Improvements in the quality of the workflow were noticed because less retakes of images are required. Films do not get lost anymore and the images can always be reported. Moreover, it was mentioned in many interviews that images get reviewed more frequently than before and there are less missed diagnosis. Specifically, the fact that images do not get lost and stolen anymore was often referred to as the most valuable aspect of the new workflow in terms of patient care. In some cases, the hard copies were also hidden by clinicians.

Residents or registrars or other clinicians would take the films because they didn't want them to get lost and then they would hide them somewhere.

In addition, staff can now retrieve images easily from the Web server without wasting valuable time on chasing film bags. Therefore, the workflow has become more agile, resulting in less physical movement. This frees up time for clinicians and other medical staff to do other things. Also, interviewees emphasized that images were often not reviewed in the past because people were too busy doing other things.

The availability and accessibility of images was mentioned in all interviews as one of the most important aspects of PACS. The system provides simultaneous accessibility to medical images and provides the means for better communication, diagnosis, and clinical treatment. Furthermore, medical staff are not pulled away from the patient. Most respondents explained that the images load within seconds and that they are usually available within minutes after they are taken in radiology. Although PACS provides earlier accessibility, the efficiencies cannot always be used to its full potential. Data suggest that it is highly dependent on how fast a patient comes back from an imaging exam, if an official report from the radiologist is needed, and the availability of doctors. Furthermore, before PACS implementation, there were often arguments between clinicians and radiologist about whether films should be up with the clinicians or in radiology for diagnosis and reporting. PACS considerably changed these problematic deliberations as the images are simultaneously available everywhere and do not impede on another's workflow.

The integration of PACS with other information technologies and information systems (IT/IS), such as the radiology, hospital, and clinical information system, further enhanced the process of accessing images, requests, and pathology reports at the same time. Also, a decline in paperwork was mentioned in the interviews. Due to the integration of PACS, the staff have access and are able to view relevant information for their practice during the whole continuum of the workflow while not incurring additional delays. The system can deal with high levels of users accessing information, but has crashed several times. This, however, has not resulted in any substantial problems according to interviewees. The system is reliable and robust enough to store more data every year and contributes to overall hospital workflow execution.

It is a very robust system, there are a lot of redundancies build into it, but it is not foolproof and depending on the problem that happens, there is a planned contingency.

There have been slight improvements in waiting time for patients to receive their exams and results due to PACS and several other improvements made in radiology. Patients do not have to wait and sit in the department for any films to be developed before they are released. Patients get their scan done and shortly after that they can be directed back to the referring clinician or wherever they need to go. With reference to these impacts, estimations vary between 5 and 10 min on average that are saved for each individual patient in radiology. This allows radiographers to do extra exams and increase patient throughput. Nevertheless, most interviewees were hesitant in mentioning specific time savings in receiving exams and suggested that most gains were due to the postprocessing capabilities of PACS.

I think the patient waiting time to receive the test may have only reduced slightly, but in terms of receiving results the waiting is certainly reduced significantly.

The patient throughput is not solely determined by the use of PACS technology. It is dependent on many factors that contribute to efficiencies in radiology and the patient's overall journey in the hospital. A common view held by respondents was that improvements in throughput were also due to improved capabilities of the different modalities and other improvements throughout the entire hospital. Also, an increase was noted in the total amount of examinations after the implementation, but the physical throughput of patients has not been improved remarkably by PACS. It has made the patient flow more patient-centered in the continuum of clinical care.

More like the possibility for throughput has been increased. I mean, I think we're still doing the same similar number of patient numbers anyway. I think it is still a fairly static number of episodes per year. It goes up by percentage every year...

Regarding this subject, the interviewees suggest that there are several impediments to patient throughput that are not necessarily related to PACS or medical imaging in general. Although workflow has been streamlined in certain areas, there are still time-consuming processes that are inevitable. For instance, there is still the amount of time to bring a patient to the medical imaging equipment and to perform the exam. Furthermore, there is also the time to bring a patient back if that would be necessary. Consequently, PACS improves parts in what was once a lengthy and inefficient process, but cannot eliminate many of the steps that make up most of the time to do an exam.

With respect to the selective codes of the four perspectives of the PACS-BSC model, results show that all aspects have been improved either considerably or in a minor way through the PACS implementation. Time-savings and image-based clinical action and patient throughput and flow showed relatively small improvements. Respondents suggested that several factors, such as the busyness of a department, the availability of operating rooms, and patient flows, have detrimental impact on these aspects.

Discussion

In this investigation, we evaluated the impact of PACS on the workflow of a large public hospital according to four interrelated perspectives of the PACS-BSC model. The framework, analysis, and results presented in this article set out the impact of PACS on workflow in a rich sense and provide insights into the complex dependency of PACS and other important aspects of hospital operations. The majority of the qualitative data suggests that improvements in workflow execution do result from PACS implementation. However, there are still some important aspects concerning the clinical treatment of patients that require careful attention, such as the earlier initiation of clinical action with PACS and patient throughput.

Reports can now be rendered in real time, which has a positive impact on both the quality of patient care and the perception of radiology services by referring clinicians. The impact of PACS on processes and the quality of work was shown to be considerable. All authorized staff can access digital images simultaneously, allowing them to make contributions in decisions concerning the proper treatment of a patient, which is an issue not previously uncovered in research. Although the process of diagnosis has been improved, no evidence supports changes in clinical decision-making and earlier initiation of clinical action, and this is consistent with the literature.^7,8 In fact, it has been argued that organizational factors other than PACS have a determining role on the initiation of image-based clinical action.⁷ To gain full advantage of PACS, the system needs to be carefully aligned with organizational dimensions as proposed by theories of alignment.¹⁸

In accordance with current research on simplification and transparency of workflow after PACS implementations,^4,5 analysis demonstrated that multiple steps have been eliminated, making the workflow as simple as possible. Moreover, the findings of this research support literature⁶ stating that a first effect that one can expect from automation is an increase in quality; it also becomes easier to manage all the different activities of a department. PACS affects the overall contribution of IT/IS to workflow by making images accessible and available at an earlier time.^7,8,19 This is shown in this research through analysis of the outcomes of PACS concerning availability and accessibility. Integration appears to be important because PACS does not stand alone. Rather, it is the interrelationship and integration with existing IT/IS that can streamline processes or can prevent the slowing of processes through parallel systems by, for instance, storing the same data. Important from an information system perspective in terms of workflow is the amount of downtime of PACS and the associated loss of records.^8,20 For a large part, it determines if processes can continue or not and therefore it is crucial that the system is robust.

Although some scholars argue that there is no significant difference in the number of examinations taken per patient between conventional and filmless radiology¹⁰, findings show that there has been an increase in the total amount of examinations over a time span of 20 months after the implementation. This allows for a better patient throughput and is coherent with prominent literature.¹¹ Analysis also shows that there have been slight improvements in waiting time for patients to receive exams and their results. Notwithstanding these improvements, the interviewees suggest that physical throughput of patients has not been improved much. Apparently, PACS does not help to resolve these issues much by only providing earlier accessibility to images. According to the patient perspective, the impact of PACS is all about patient centeredness in the continuum of clinical care. The workflow has been improved in certain areas, allowing patients to flow more smoothly through the entire patient care delivery process and the main challenge should be to minimize unproductive time and reach higher utilization levels with all the modalities to optimize patient throughput.

Conclusion

The obtained selective codes from the data analysis fit effortlessly on each of the proposed perspectives of the PACS-BSC model. Although this model is applied to a single case, the codes do validate the expanded model. The conclusion is that the PACS-BSC model is fundamentally suited to evaluate PACS implementations in public hospitals from different angles and that the inclusion of clinical and not-for-profit goals in the strategy may make the evaluation more relevant. Although the inclusion of qualitative themes rather than quantitative performance measures suggests different research methods, the model remains consistent with Kaplan and Norton's intention of providing a single report to evaluate outcomes relevant to the organizational strategy. Although various efficiencies have been reported in this study, important issues still remain unanswered. Because PACS is a system designed to streamline operations through the entire patient care delivery system, one would expect that it significantly makes a difference in terms of throughput and clinical action as well. Theories on business/IT alignment, organizational fit, and adoption of IT/IS can help understand why certain key elements in clinical practice have not been realized.¹⁸