Back to the vinyl age: a narrative report of a total computer blackout at a large university medical centre

Abstract

We report here our experience during a total blackout of all computer systems at a large-scale medical university centre lasting for almost 2 days, affecting not only the hospital information system but also all picture archieving systems, access to laboratory data, office, and email software and even the personnel’s ability to log into their accounts. While initially threatening, staff quickly adapted to the situation and the promise of ‘digital health’ to enable more time for the patient and improve communication, was essentially fulfilled when our computer system was totally down. Based on our experience, we recommend an involvement of health professionals and their medical societies in every step of the digital transformation to accomplish this mission in a responsible, safe, and human-centred way.

The technical perspective

It was Thursday, 29 August, which was a quite hot, late summer day in Heidelberg, Germany. At 5:45 pm, there was an electronic new sticker announcement about upcoming maintenance work on the medical computer systems. The IT work started as planned at 6:00 pm. Only 1 h later, one of two major storage clusters no longer entered the boot sequence after being updated. Unexpectedly, the redundant system did not take over and also failed. As a result, many user applications started to fail, including Windows Office suite, intra- and internet access, the hospital information system, all connected services including lab results and digital medical dossiers, the central PACS service, and login servers for user accounts.

At 7:00 pm, the hospital’s central emergency task force was informed and a support call to the manufacturer of the storage system was initiated. Starting from 07:30 pm, several troubleshooting attempts following the manufacturer’s standard support recommendations were unsuccessful and all available administrators of the hospital’s IT team came on site. At 10:00 pm, the update of the second cluster was completed according to schedule, and the joint system was still non-operational.

Friday 30 August, 00:00 am, the support attempts were still unsuccessful after summoning the manufacturer’s global crisis team and establishing a permanent web meeting with the hospital (until 8 pm). At 04:45 am, a new sticker message to all users indicated that far-reaching IT disruptions would continue and most likely escalate. At 12:00 am, the manufacturer identifies a specific hard drive model to trigger the storage server issues, an error that did not occur with the previous firmware version. In order to get the system operational again, the problematic hard disks were removed from the raid system and carefully replaced to allow for the restoration of raid integrity. Replacement hard drives were shipped from multiple warehouses in Europe and the USA and arrived on Friday afternoon. A technician from Sweden arrived and stayed on site from Saturday to Sunday. Altogether 100 hard drives originating from 780 panels were replaced.

At 07:00 pm, the first successful booting of the affected system was achieved, and all application servers started after restoring databases from the backup systems. At 08:40 pm, employees were informed via an electronic message that all basic systems were active again. Saturday 31 August, at 08:00 am the last problematic hard drives were exchanged, mirroring was reconstructed, the integrity of the databases was tested, and staffs were informed at 01:00 pm that almost all systems were available. Finally, a cyber-attack was ruled out and the severe incompatibility of the storage system firmware with distinct hard drive models documented. The timing of events is shown in Figure 1.

Figure 1.

Timing of events that we documented during the course of the computer blackout.

The human perspective

Despite increased loading times for medical records and sluggish response times of all computer systems, our doctors in the night shift felt still relaxed since the hospital did never experience a complete failure of the medical information system. On Friday morning, as usual, our department was crowded with patients—doctors and nurses in the morning shift tried to login to their systems. The information on the total failure from the night reached the last of us just minutes after 8 am. It took several reboots until the employees became aware of the full extend and severity of the situation. The senior physicians of the department met in an ad hoc meeting and immediately connected to the central emergency task force, which gathered all information centrally, by phone. Email and intranet were not operational at all. Handwritten papers with essential phone numbers for querying lab results, manual documentation sheets, and other important information were photocopied and distributed. Especially in the units handling emergencies and invasive procedures, the suggested measures were quickly stablished and resulted in an operational chest pain unit and the catheter lab. In the latter, the chalkboard, being a rare relict of the analogue world sitting next to the two 55″ LED screens normally used for patient scheduling and thus unused for the past 15 years, was put to use and the technicians quickly established their own operating procedure for the coming hours of uncertainty.

As senior physicians, we approached all units personally and offered information and help in any possible way. In the coming hours, we spent time on the intensive care units, the chest pain unit, the catheter lab, the normal wards, as well as the echocardiography and outpatient units. Our staff treated more than 150 cardiovascular patients on this day and after the initial uncertainty, the entire team experienced some unexpected and positive surprises:

• Instead of sitting in their offices and looking at the ubiquitous digital information from past and present stays of their ID-tagged and real-time monitored patients, hunting for the next piece of ‘valuable’ information in the depth of the computer systems, doctors talked to their patients, made comprehensive anamneses, and conducted thorough physical examinations. Explaining the situation regarding the blackout to the patients quickly turned into content rich communication about what was relevant for the patient. Instead of asking text behind checkboxes and spending hours on putting Barthel indices and alike into documentation systems, nurses and doctors strikingly enjoyed the effective use of their and their patient’s time.

• The forced untethering from the digital world seemed to have a liberating effect on our colleagues. They did not only spend more time with their patients, but we have seen them discussing individual patients in-depth together. For many, it was surprising that the time was suddenly sufficient for all those interactions.

• Since lab tests needed to be acquired via phone calls to the central lab, tests were indicated after thoughtful judgment of the clinical situation. This led to an assessment of 156 troponin tests, whereas a mean of 251 tests is performed on a normal Friday in our university hospital according to the central laboratory’s records.

• Ultrasound and echocardiography reports were written on a blank sheet of paper. The reports were thoughtful and based on the doctor’s own analysis. The well-known effect of ‘copy and paste’ of previous medical records¹ was gone, which reminds us of the quote from Douglas Adams: ‘The most misleading assumptions are the ones you don't even know you're making’.²

What did we learn?

The promise of digitalization and the introduction of artificial intelligence into medicine is a mesmerizing prospect.³ At the first European Society of Cardiology Digital Summit in October 2019, just weeks after this unintended experiment, we discussed how to make the workforce of tomorrow ready for the digital future. Concerns from many discussants were pressing that increased digitalization also leads to increased burn-out rates of nurses and doctors. Doctors in hospitals nowadays spend more time in front of screens than with their patients^4,5 and medical IT is dominated by software solutions that are neither user-friendly nor made to improve the work of the doctor—they are primarily made for documentation and accounting. The often-heard promises of ‘digital health’ leading to improved patient evaluation, easy reporting, more time, and better communication, were essentially fulfilled when our computer system was totally down (Figure 2). Those 2 days were the only days in our whole carriers as doctors without any computer system and—sadly?—they most likely will remain the only two ‘analogue days’. We do not want to hide that some departments of our hospital were running in an ‘emergency mode’, e.g. the elective surgery program was skipped. Therefore, the patient volume during these 2 days might not reflect a ‘normal business’ day in every department. A prolonged blackout would clearly have led to more severe limitations in treating our patients and the financial burden of the computer breakdown was certainly tremendous, as several examples from hospitals being victims of crypto-Trojan attacks have shown. Such a prolonged breakdown would also lead to a risk of delayed diagnosis and treatment, yielding a considerable risk of harm. However, through the lens of our professional life as medical doctors, many advances of digitalism we both so much belief in are not as obvious any more after these 2 days.

Figure 2.

Positive and negative aspects and consequences, which were noted during the total blackout.

Therefore, we strongly encourage the medical community to call for a thoughtful, user-centric transformation towards digital health, bearing in mind the requirements of doctors and their patients. Based on this experience, we recommend an involvement of health professionals and their medical societies in every step of innovation and implementation of digital technologies in hospitals. One even might call for health professionals specifically trained in digital health technologies to foster this most critical transformation of medicine in its history. These specialized physicians might help to facilitate broadly applicable solutions and also adapt user centric innovations at their local hospital as well. Beyond that, we need to warrant the integrity and safety of computer systems involved in treating patients by appropriate funding lines for independent research. Maybe we need experiences like the one reported above to reflect our own progress, according the quote: ‘Sometimes when you innovate, you make mistakes. It is best to admit them quickly, and get on with improving …’ (Steve Jobs).

Funding

B.M. is supported by the Deutsche Forschungsgemeinschaft (DFG ME 3859/4-1), the German Centre for Cardiovascular Research (DZHK), Informatics for Life (Klaus Tschira Foundation), the ERA-CVD network DETECTIN-HF and the Else-Kröner Exzellenzstipendium.

Conflict of interest: B.M. is scientific advisor of Fleischhacker GmbH and received research support by Siemens AG and Apple Inc.

Data availability

No new data were generated or analysed as part of this paper.