From Prototype to Production System: Lessons Learned from the Evolution of the SignOut System at Mount Sinai Medical Center

Abstract

Approaches to the development of information systems in large health care institutions range from prototyping to conventional development of large scale production systems. This paper discusses the development of the SignOut System at Mount Sinai Medical Center, which was designed in 1997 to capture vital resident information. Local need quickly outstripped proposed delays for building a production system and a prototype system quickly became a production system. By the end of 2002 the New SignOut System was built to create an integrated application that was a true production system. In this paper we discuss the design and implementation issues in moving from a prototype to a production system. The production system had a number of advantages, including increased organizational visibility, integration into enterprise resource planning and full time staff for support. However, the prototype allowed for more rapid design and subsequent changes, less training, and equal to or superior help desk support. It is argued that healthcare IT systems may need characteristics of both prototype and production system development to rapidly meet the changing and different needs of healthcare user populations.

Introduction¹

Many information systems in the health care domain were initially designed to service local settings and limited user populations. In addition, many Informaticists have constructed prototypes to test research ideas in applied settings such as hospitals and clinics. However, these systems have often been drafted into production due to meeting a critical and clearly defined local need and/or lack of local funding to build a production version or buy an alternative. Such systems have subsequently been scaled up to service a larger number of users and a wider range of work settings. For example, BICS was built by Informatics to conduct research and meet local needs¹. Many would argue it is superior to any commercial product. However, replacing it, would be prohibitively expensive as a commercial replacement would cost millions and only save the hospital $100,000 a year². Similarly, Beth Israel's Outpatient Electronic Medical Record³ which many would argue was superior to any commercial product of it's time would cost millions to replace⁴

The literature in software engineering has described a number of functions for development of prototypes. Prototypes, can be defined as limited versions of a system with at least some level of functionality ^5–7. They are used at various points in the software development life cycle (SDLC) for various purposes. Prototypes may be designed to determine what user needs are from which a requirements specification for a production system can be determined, after which the prototype may be discarded (these are known as “throw away prototypes”). Other prototypes, known as “evolving prototypes” may be continually refined until they eventually become production systems. In the IT healthcare field, a service world, prototype may mean ready to code and roll out a production system, although it may be limited in terms of scope and type of users. As a result, confusion often exists regarding the purpose and development of health care information systems. For example, the Old SignOut System at Mount Sinai Medical Center was designed to capture vital resident information. Old SignOut had the reliability, staffing maintenance and support characteristics of a prototype, the user population and lifespan of a production system, but coding that would not allow it to be rolled out across the Enterprise.

Background

In 2000 a decision by IT was made to build an integrated enterprise version of the SignOut System called New SignOut. The New SignOut System was designed from the beginning to be a production system. In migrating from our prototype system to an integrated production system we learned a number of lessons that will be described in this paper and which have important implications for the development and implementation of systems in health informatics.

Initial Development and Design Process

The original SignOut System (i.e. the prototype “Old SignOut System”) was constructed in 1997 by one of our then Informatics fellows to meet the needs of medical housestaff⁸. Housestaff SignOut's have been successfully computerized before ^{9, 10}. However, Old SignOut focused on capturing and storing clinical information about the patient’s hospitalization directly from inpatient providers during the hospitalization. More importantly our system captured and stored inpatient provider information during hospitalization. After the patient is discharged, an interim discharge summary that contained clinical and provider information from the last signout (day of discharge) was accessible via hospital Intranet.

At the time of development of Old SignOut, inpatient housestaff at the end of every working day would signout to the on-call team. Prior to development of Old SignOut in 1997, signout was accomplished by using word processing programs such as Lotus™ Ami Pro 3.0 or Word Microsoft™ Word 6.0. Residents used word processors to make patient lists and include information that the on-call team needed to be aware of for the night or the weekend. Information that the residents supplied for signout included relevant histories, medications and other issues of concern. The signout was saved on a local computer in the on-call room and was updated everyday. When a patient was discharged from the hospital, the patient’s name, providers, and clinical information were deleted from the signout by housestaff and forever lost.

Before software development was initially undertaken, we identified key design objectives for old SignOut in 1997: 1) Provide functionality equal to the previous signout done using a word processor 2) Capture clinical information and provide this information immediately in real time 3) Create an interim discharge summary that accurately identifies inpatient providers and provides some clinical information. Development of the system was begun in March 1997 and was ready for pilot in July 1997. User input was direct to the developers who were practicing clinicians and therefore would encounter users in the hospital. Changes were made in a matter of days to weeks while changes that required integration with hospital systems were not done. For example, users wanted medications from the computerized physician order entry system (CPOE) to be added to SignOut. At time of development the hospital system was a distributed network, there was no repository and IT had little experience with developing applications. However, later on IT linked up test results to the Nephrology version of Old SignOut using an interface gateway and an FTP file.

The Old SignOut was developed using HyperText Markup Language (HTML) for form development. Stormcloud Development Corporation's WebDBC 3.0 connected the forms to a Microsoft ®Access 97 Database. The system was accessed via Mount Sinai's Intranet by using Netscape 4.x and later Internet Explorer 5.x. The system ran on a Microsoft ® Windows NT Server 4.0 Service Pack 3 subsequently upgraded to service pack 5 and to Windows 2000. Secure access was provided for by a two-password identification system, and the system was only accessible via the Intranet. Server backup and maintenance was done on server residing next to developer. The tape drive used for backup eventually failed and was replaced.

Evolution of the Initial SignOut System

The system grew over time to encompass all Medicine housestaff services, briefly Pediatrics, Consult services of Medicine, Geriatrics and Renal, Renal HD service, and ADS (Attending Directed Service) NP's. As volume of users grew customization by service was requested with increasing frequency. With the exception of Medicine's 3 General Medicine Services, SignOut's were built on a service by service customization. The Pediatrics implementation failed as the pediatrician builder had neither sufficient time nor resources to maintain the system. By July 2002, the Old SignOut System had generated interim discharge summaries on 16,976 patients eventually making it far from a prototype.

As the user base of Old SignOut expanded, the system began to crash with increasing frequency. WebDBC opened one session per user, occupying increasingly valuable system memory, and closing the session resources long after user logged out. WebDBC, whose parent company had gone out of business, was replaced with Cold Fusion 3.x, which stabilized the system. SignOut was down during server backups and antiviral scanning. The Help Desk was manned mainly by the system developers. Sinai's help desk would receive some calls and forward those calls to us.

Extension to a Full Production System

In 2000 it was decided to reengineer the Old SignOut System and the New SignOut was born. It took 1 year to design and translate specifications from Old SignOut into New SignOut and design needed enhancements and features. Unlike the development of the Old SignOut, full time IT staff were assigned to the New SignOut project (although they were not dedicated exclusively to the project). The expectation was that IT staff would leverage existing skill sets and knowledge and use them to build New SignOut as a production system.

Based on the lessons we learned from Old SignOut, the following requirements were set up using a formalized approach to requirements specification: 1.) Enterprise ownership of New SignOut as this would be critical to resourcing and planning. 2.) Significant improvements to security including automated creation of user id and password, password expiration date and an extensive audit trail. 3.) Remote accessibility 4.) Consistent spelling and use of attending names. 5.) Automated tracking of coverage schedules. Manual maintenance would significantly limit extensibility to other services. 6.) Automatic updating of monthly service (assigned) attendings as each month the service attendings would predictably change. 7.) Resident log-in preset to "their signout". In the Old SignOut System, for the entire service month, each time residents logged on, the resident would be presented with a service drop down menu, and would then have to select as service and team. 8.) Utilization of interim discharge summaries. The Old SignOut created interim discharge summaries but required users to go to the Informatics web site, select and click on a special link to view. The interim discharge summaries were predictably under utilized. 9.) New DocFind. The purpose of the Old DocFind was to facilitate identification of and communication with housestaff by identifying patient specific housestaff providers along with their beepers from SignOut data⁸ 10.) Extendibility to multiple services. Our goal was to build one (system) used by many services. 11.) 24x7 help desk support that addressed and routed all questions.

Assessment of the Production System

Several months were spent doing extensive testing and bug fixes. Some features were completed during this time after more input from the original developers of the Old SignOut. However, design was frozen early through testing to guarantee release date of Dec 2001. A significant change that occurred in process was the developers were no longer directly receiving input from the users. The Informaticists had become the design team building on lessons learned and the developers had gone from clinicians to programmers and data base administrators. Also of note that during New SignOut construction was that significant work and resources were committed to EDR, the clinical repository, at both Sinai and NYU. The lessons learned from EDR would be translated into later SignOut development.

During the pilot all bugs were fixed in a timely fashion. While the development staff was more than accommodating, institutional priorities and time allocation, resulted in user enhancements being done when possible but not as rapidly or as frequently in the Old SignOut System. Significant enhancements in either performance or function occurred in a time period ranging from weeks to months.

Comparison of the Systems

The front end of the New SignOut System is written in Java and JSP and resides on a Solaris web server. The backend database is an Oracle 8I database residing on a R6000 AIX machine. In contrast the Old SignOut System was written in HTML, used WebDBC to communicate with the database Microsoft Access and resided on an NT server. When a patient is added to New SignOut demographic information including name, age, sex, medical record number, and admit date is transferred to New SignOut from EDR which has information on 236239 hospital/clinic patients. See Figure for architecture. The most noticeable change in architecture are integration with both hospital (i.e., credentialing and EDR) and external system Amion (see figure to the left). The contrast in architecture can be seen in the figures above.

Using a pass through of user id and password from the CPOE, Eclipsys 7000, provided security. Not all order entry system users have access to SignOut. When the password expires in CPOE, users must log into CPOE and change the password. Updating requires 1–2 hours. Warning are provided days in advance but sometimes ignored.

SignOut continued to be an Intranet accessible application. As a result, New SignOut was accessible remotely as result of institutional development of Extranet with VPN security.

The New SignOut System data structures were designed to be able to receive data from a commercial oncall and coverage system, Amion. Amion is a web-based scheduling system used by residency training programs nationwide. Amion gives both the daily oncall schedules for each service including physician name, training level (e.g., PGY) and beeper number for each service as well as listing all clinical assignments for each block rotation. SignOut integration with Amion was initially difficult. Pediatrics was already using Amion. Medicine required many hours of developing templates to use Amion as the Medicine housestaff program is complex with a large number of housestaff.

The New SignOut has been widely successful with discharge communiqués, the successor to interim discharge summaries. As of end of January 2003 there are 7926 patients with communiqués. Integration with EDR has resulted in communiqués being visible to all in EDR and 13,000 accesses reflect the change from looking in yas (yet another system) to the central EDR. Changes resulting from New SignOut include:

1. Revamped DocFind. This has taken longer than expected as institutional resources in short supply and redesigning it has been proven to be more complex as it must be extendable. The result is a feature that was critically lauded and used in the old system, though only for Medicine, has been out of service for over a year.

2. Extendibility to multiple services hasn’t improved as much as mandated. Services are being asked to use the system as is or fund enhancements. We suspect that many services have agreed to use the system as the overall functionality and integration has outweighed disappointment of inability to customize. However, lack of customization has slowed the progress of the system particularly to consults.

3. 24x7 help desk support has occurred but despite attempts, including creating an escalation manual, most calls are still forwarded to us.

4. Speed. The production system was initially slower and generating some complaints. Each access of the SignOut patient list resulted in a call to EDR for an update of patient demographics. Since patient demographics rarely change, this updating was stopped. New SignOut is now as fast as Old Signout. Bed and floor are still updated with each patient list access.

5. Reliability. New SignOut was initially buggier than Old SignOut. However, New SignOut had many new enhancements and features added. More disconcerting were nighttime database/server failures in the first few months of operations. Little could be done to identify and fix the problem often till the following day. Hospital outages of clinical systems now affected New SignOut whereas Old SignOut was immune to these. In contrast, Old SignOut’s outages resulted from WebDBC.

Lessons Learned in Scaling Up

Overall, the construction and transfer of services from Old to New SignOut has been a positive experience. The progress on the user interface and functionality has been incremental and impressive. The integration, which supplies patient demographics and updates bed and floor, have been well received. Amion is becoming part of the culture in Medicine.

The increased organizational visibility and integration into enterprise strategic planning has lead to heavily utilized discharge communiqués in EDR and integration with central sources of clinical data. The breadth of users despite limited customization is impressive. At it's height the Old SignOut serviced Medicine and Geriatrics and affiliated services. New SignOut services include Medicine, Pediatrics, and ADS Servicesx2. Neurology, Psychiatry and Hospitalists are to follow.

Training needs have increased due to increased feature set. Less training was required with the Old SignOut System because the feature set was less ambitious and the thought was we could always add more features on the fly. There were almost no “how to use questions” with the Old SignOut while such questions still routinely occur with New SignOut.

The 24x7 help desk has been disappointing. Some of this is general help desk performance and some is flexibility lost with production versus prototype. In a prototype system one can rapidly go in and fix the system. The production system is supported by a larger group that unfortunately is not always available.

The change in roles created another layer of personnel between user and system. In Old SignOut the design/development team were clinician Informaticists who would receive user input and translate directly into development. In New SignOut the designer/developers became the design team. User input was still received but the design team was no longer responsible for making the changes. The development team at times had difficulty determining if suggestions for enhancements came from users or design team interpretation of users.

The New SignOut development team demonstrated a high level of professionalism. Bugs were addressed efficiently, the cause of bugs found in an expedited fashion, and any changes needed were made. The development team took great pride in meeting specifications and making them work. The development teams also received conflicting priorities from the design team their own managers and institutional resourcing priorities.

The prototype (Old SignOut) allows for more rapid design changes such as adding services requiring customization, rapid development of add-on features such as Old DocFind. Full time development and standard tool sets avoid using tools that become extinct.

The Old SignOut system was initially reliable with a small user base but became less reliable as the software was unable to support a growing user base. In contrast, the New SignOut had unanticipated reliability issues early on, server problems, but became more stable despite increasing numbers of users. New SignOut was dependent on quality of hospital data, and there were unanticipated data quality problems.

The costs involved in developing and moving to the New SignOut System were in the order of 50 to 100 times the cost of the initial prototype system (however, the cost of production system was never directly billed).

Conclusion

In summary, the initial prototype Old SignOut System allowed for rapid rollout due to less training overhead, rapid development, and easier to support help desk. However, the New SignOut production system had an extended feature set, integration with other systems eliminating duplication of data, centralized access of discharge communiqués, and a wider breadth of services using system.

Healthcare IT needs the rapid design and help desk support of the prototype to address and define the requirements of clinical users but also needs the reliability and both technical and organizational integration that a production system brings. In moving from an initial prototype system to a production system we identified the following factors and issues that need to be taken into account: 1) transfer of knowledge, skills, roles and responsibilities within the organization 2) development of more extensive training and support infrastructure 3) requirements gathering that build on the experiences of previous, more limited system versions. In addition, by initially focusing on development of a more limited prototype system and then scaling up or reengineering, we have found that issues related to meeting user needs may be more easily addressed. These factors may have implications for understanding the success of many scalable prototype systems (which were subsequently drafted into production use) in relation to larger production and commercially available health care information systems. Furthermore, such scalable systems may end up being in keeping with the workflow, systems and needs of the local clinical environment. In conclusion, our experiences indicate that there may be a needed blurring of the distinction between traditional prototype and production system models and processes.