PACS Bypass: A Semi-automated Routing Solution to Enable Filmless Operations When PACS Fails

Abstract

In the filmless imaging department, an integrated imaging and reporting system is only as strong as its weakest link. An outage or downtime of a key segment, such as the Picture Archive Communications System (PACS), is a significant threat to efficient workflow, quality of image interpretation, ordering clinician’s review, and ultimately patient care. A multidisciplinary team (including physicists, technologists, radiologists, operations, and IT) developed a backup system to provide business continuity (i.e., quality control, interpretation, reporting, and clinician access) during an extended outage of the main departmental PACS.

Background

In many filmless environments, the downtime procedure for a Picture Archive Communications System (PACS) outage may rely on a return to film. At our site, this would require approximately $45,000 per day in film costs, a manual distribution network that no longer exists, printer maintenance and capital costs, view box availability, and alternate quality assurance (QA) processes. Our situation is likely not unique in some of these challenges. In addition to the prior point, film suffers from significant limitations: access is restricted to a single point of use, no image processing or advanced viewing tools can be used, and one lacks early viewing capabilities for clinicians (since the film cannot be shared outside the department until the radiologist interpretation).

These realizations drove our search for an alternate strategy that could maintain a filmless radiology practice in spite of a PACS outage. A multidisciplinary downtime team (including physicists, technologists, radiologists, operations, and IT) was formed. Early on the team defined the following requirements:

• Include seamless integration with other electronic health records and image management applications

• Use existing hardware and software where possible

• Provide simple and rapid switch over to and from the backup system

• Leverage tools already familiar to the radiologists and technologists

• Be easy to use and allow radiologists and technologists to efficiently perform their work with minimal interruption.

Several options were considered:

1. Preloading PACS workstations with dual viewing applications: the normal PACS viewer and a backup viewer supported on independent servers. Upon a PACS outage, one would have the radiologists switch to the backup viewer

   • Pros: During a PACS outage, radiologists could continue to use the same workstation, with calibrated displays, that they used moments before
   • Cons: A separate, independent client–server system needs to exist that can be leveraged in this manner.
2. Hardware redeployment: replace broken PACS workstations with functional workstations from non-busy areas.

   • Pros: Technologists and radiologists use familiar applications
   • Cons: While adequate for local workstation breakdowns, this does nothing to maintain business continuity if the central servers are compromised.
3. System hardening/outage prevention: clustering of the PACS servers, using mirrored disks on the PACS workstations/servers, etc. [1].

   • Pros: Reduced outages due to hardware failures
   • Cons: Does not help if outage is due to a software upgrade, software defect, or human error
4. Hard copy/manual operations: the manual printing and distribution of hard copy films.

   • Pros: Better than nothing
   • Cons: As described above, this was evaluated and found to be inadequate for our high-volume practice. Costs were prohibitive and required system infrastructure was no longer in place.

We also reviewed the thoughts of others in this area, and found they had similar conclusions [2, 3]. Armed with this list of potential strategies, we began to examine the options with regard to the surrounding digital environment.

For the reasons seen above, scenarios 2 and 4 were quickly dismissed. Scenario 3 is already present at our site; that is the Production PACS servers are all clustered in dual data centers in a high availability (HA) configuration. However, this is not sufficient to assure uptime, contrary to what many believe. Unplanned system outages can be traced to three causes: hardware failures, software failures, and human error [1 op cit]. An HA configuration does nothing to address the last two. For instance, if there is a software defect that causes a file system to be corrupted on one server, running the same software on a backup server will also cause corruption (i.e., three failures at our site in the last month were not resolved by resorting to the backup servers). To avoid this, one has to have independent execution paths on different software. This is what scenario 1 leverages.

At our institution, we are fortunate to have an enterprise image archive and an enterprise clinical viewer [4–7]. Hence, the main objection in scenario 1 (costs of additional servers and other infrastructures) was not an impediment in our case. In normal operations, unread exams in Radiology are quality controlled (by technologists) and interpreted (by radiologists) on the PACS. Only when there is a final report in the Radiology Information System (RIS) can most clinicians view an exam. However, there is an ability to view exams in the clinical viewer before a final report. Such capability (known as “early viewing”) is usually only exercised in the ER and OR, and nowhere else during normal operations. However, our downtime team began to consider altering this behavior. Specifically, we wondered could we, in the event of total PACS failure, reroute the department’s output around the PACS to the clinical viewer. The next section describes how this question was turned into a downtime strategy.

Methods

Our institution employs PACS pre-processors (P3s) between the imaging modalities and the PACS. These systems apply modality and vendor-specific image processing to images before the images reach the main departmental PACS. For example, MR images have window/level values set automatically before they reach the PACS. In a similar manner, computed radiography (CR) images have custom lookup tables applied. The result of these operations saves the radiologists considerable time—image appearance is predictable and optimized largely without human adjustment. A simplified overview of normal data flow in the Radiology and Enterprise imaging systems is shown in Fig. 1.

Fig. 1.

A simplified view of normal data flow for clinical images at our institution. Modality clouds send exams to a load balancing switch that distributes those exams to the PACS pre-processors (P3s). In normal operations, the P3s perform their image processing and then route their output to the PACS. After there is a preliminary (or sometimes final) interpretation, the exams are sent to the institutional image archive and its attached clinical viewer for clinician use

The P3s are the keystone in the PACS bypass strategy. By default, any P3 can accept any image type, but they have routing rules that map exam outputs to PACS servers based on the source imager’s modality and location. For example, CR from Hospital A may go to PACS server 1 (and so on). The servers obtain these routing rules when they boot up by copying them from a file server. If one were to substitute different rules on the file server, a simple reboot of the P3 would alter its routing behavior (see Fig. 2).

Fig. 2.

At boot time, P3s obtain their routing rules from a file server. During a PACS outage, bypass tables are substituted for the normal tables, and the P3s bypass the PACS to send straight to the institutional image archive and its attached clinical viewer

From Fig. 2, one can see that swapping the default routing tables for the PACS bypass tables results in radiology exams going directly to the institutional archive and clinical viewer. Fortunately, access controls in the clinical viewer allow most clinician access to be gated (except OR/ER) until at least preliminary interpretations are available.

Another useful feature of the P3 is that it spools exams if they cannot be forwarded to the intended target. Hence, when PACS fails and goes offline, the P3s can spool until the bypass is initiated, and then those exams will flow to the Archive. The one tricky issue is what is done about the exams that were interrupted in transit. To address that, IT support examines the error logs of the P3s and notifies the technologists of failed transfers. The technologists then resend those cases from the modality.

Once the decision has been made to execute PACS bypass, the normal workflow is altered. Exam QA (normally done by technologists on PACS) has to be performed on the modality itself. Also, the radiologists, while still being able to work on their PACS workstations, will need to shut down the PACS software and start the clinical viewer. This is assisted via desktop icons that coordinate the conversion from normal to downtime applications. Meanwhile, a departmental web page (launched on login) tracks system status and advises staff on downtime procedures (Fig. 3).

Fig. 3.

A status web page informs radiologists of departmental and enterprise system status and provides links to downtime plans

Once the PACS is back online, communications from department leadership once again synchronize halting image sends from scanners to allow the P3s to be quiescent during their reboot and resumption of normal routing. Once they are returned to service, the practice resumes normal workflow.

The decision to execute a PACS bypass is coordinated. The downtime team met for over a year to interview all stakeholders, document normal and bypass workflows, and work with lead technologists to draft area-specific downtime procedures. The final policy includes provision for:

1. Local command centers (one at each hospital and major clinic building)

2. Local downtime plans (available in electronic format on file servers and also in hard copy)

3. A departmental command center (DCC), staffed by the department chair, administrator, each division head, lead technologists from each area, and IT and medical physics representatives

4. Communications distributed via text pagers, email, and a status web page

Results and Discussion

When outages have occurred, the chair (or administrator) confers with IT staff to ascertain whether the outage is likely to exceed an hour. If it likely will, a DCC is convened and input sought from all attendees before the decision is taken to execute bypass. Once the decision is made, a time is agreed upon to allow all scanner sends to cease, then the P3s are rebooted to the bypass configuration. Upon the P3s coming back online, imaging areas are alerted that they may resume scanning, using their area-specific downtime methods.

In bypass mode, technologist QA workflow is altered. Normally, images undergo QA on PACS and the status change of the exam triggers the PACS to send a message to the RIS that the exam is complete. This message in turn allows the exam to show on the radiologist’s interpretation worklist. However, when PACS is down the technologists must QA on the modality and manually declare the exam “complete” by logging into the RIS.

The radiologist also experiences some changes. While still on the same workstation and displays, they must now shut down the PACS application and open the clinical viewer. The RIS-driven interpretation worklist and integrated speech recognition dictation system (which allows the user to click on an exam and have it, and its’ comparison exams(s), display on the PACS) is still available, but it cannot drive the enterprise clinical viewer. Hence, the radiologist must manually verify that the exam chosen in the RIS-based interpretation downtime worklist and integrated speech recognition dictation system matches the exam manually and independently displayed in the enterprise clinical viewer.

To date, the PACS downtime bypass has been executed once during an unplanned outage and functioned well. Additionally, the process has been selected as the method to be used for planned outages lasting more than 1 h and has twice been used for this purpose. Lessons learned from use of the system include:

Before Downtime

Communication

A strong, robust communication process must be utilized to convey the outage to the technologists and radiologists. During normal business hours, this could effectively mean communication of the downtime process to approximately 400 staff in our practice. The communication process includes utilizing a department web page and banner; email notifications; display pages; and pre-designated radiologist point persons for a specific modality, division, and location.

Training

Although the downtime process is straightforward and easy to implement, it requires training which can be difficult to accomplish in a large busy practice. Therefore, it is imperative to conduct drills to familiarize the technologists and radiologists to this process. At the very least, all personnel involved in the downtime need to know where to find the information on what to do when a PACS outage has occurred. Detailed instructions including screen shots on how to launch the enterprise viewer have been made and place on a downtime web page. Similar documents have been made for the technologists.

During Downtime

Consistent Image Interpretation and Archival

Some care must be taken to assure that the images QA’d by the technologist are all interpreted, and medico-legally we require that all interpreted images must be archived. There is some slight risk that the technologist may send additional images (e.g., repeats) after the radiologist has interpreted the initial set. Without the workflow exam status checks present in PACS (e.g., exam ordered, performed, QA completed and interpreted), it can be difficult to catch and interpret these delayed additional images.

Paper Trail

Technologists are told to save the RIS exam sheets that accompany the patients during their procedure for subsequent clean up and validation after the outage is over.

After Downtime

Downtime Exams Are Not Sent into PACS at PACS Restart

Exams processed during downtime are not sent to PACS unless they happen to be historical comparisons for exams at a later date in which case they follow the normal pre-fetch rules for comparison exams. This is important to realize because if they were sent to PACS after its restoration, the RIS messages to the PACS to archive an exam (issued at exam report finalization) would have already been fired. Hence, those downtime exams would never “see” an archive message, and the PACS would never be able to clear them from its short-term cache. By never treating them as “unreads,” the PACS can process them later just as it would any pre-fetched exam.

Radiologist Checking of Resident Interpretations

The two planned outages have occurred during off hours which have compelled us to address the challenges faced at a teaching institution. The examinations read by the residents at night need to be overread by the morning staff (who are usually not the on call overnight radiologist) utilizing the same enterprise clinical viewer that is used during the downtime. It is critical that the overnight outage be communicated to the morning staff since this is where delayed staff interpretation may occur (i.e., if morning staff expect to see the resident cases on PACS).

Clean Up

The paper trail of the examinations performed by the technologists during the outage assists in validation of the completion, interpretation, and archiving of the studies processed on the clinical viewer during the bypass process.

Conclusions

The electronic PACS downtime system described herein minimizes workflow interruption. The semi-automated PACS bypass allows for a direct feed of images to the enterprise image archive and clinical viewer. This in turn enables radiologists, with a few mouse clicks, to continue their work during an extended PACS outage. The radiologists continue to use the RIS-based work lists (albeit altered, downtime views) to load and view new and old exams utilizing the existing PACS workstation hardware. Reporting and dictation continue on existing, familiar systems.

There is some impact to the technologists’ workflow and quality assurance processes, but it is less than if there was no system in place to handle an extended PACS outage. The impact to the referring clinician is negligible. During an extended PACS outage, a significant radiology department examination backlog is avoided, the quality of image interpretation is maintained, and ultimately timely patient care is preserved.