Establishing a Health Information Technology for the Vaccination of National Institutes of Health Staff

Abstract

Introduction:

Healthcare organizations faced unique operational challenges during the COVID-19 pandemic. Assuring the safety of both patients and healthcare workers in hospitals has been the primary focus during the COVID-19 pandemic.

Methods:

The NIH Vaccine Program (VP) with the Vaccine Management System (VMS) was created based on the commitment of NIH leadership, program leadership, the development team, and the program team; defining Key Performance Indicators (KPIs) of the VP and the VMS; and the NIH Clinical Center's (NIH CC) interdisciplinary approach to deploying the VMS.

Results:

This article discusses the NIH business requirements of the VP and VMS, the target KPIs of the VP and the VMS, and the NIH CC interdisciplinary approach to deploying an organizational VMS for vaccinating the NIH workforce. The use of the DCRI Spiral-Agile Software Development Life Cycle enabled the development of a system with stakeholder involvement that could quickly adapt to changing requirements meeting the defined KPIs for the program and system. The assessment of the defined KPIs through a survey and comments from the survey support that the VP and VMS were successful.

Conclusion:

A comprehensive program to maintain a healthy workforce includes asymptomatic COVID testing, symptomatic COVID testing, contact tracing, vaccinations, and policy-driven education. The need to develop systems during the pandemic resulted in changes to build software quickly with the input of many more users and stakeholders then typical in a decreased amount of time.

Vaccine Clinic Directive

The National Institutes of Health's (NIH) executive leadership identified the need to vaccinate the organization's workforce the beginning of December 2020. The CDC provides guidance regarding vaccination of healthcare workers.¹ The effort at the NIH required coordination across many divisions and staff along with the creation of a collaborative program team. For the initiative to succeed, the team ensured there was active and visible senior leadership support; an organizational culture focused on worker safety; staff understanding and acceptance of the rationale for vaccination; availability of experienced staff to conduct vaccinations; and systems to support the scheduling, ordering, administration of the vaccine, and provision of data to manage all associated processes.

Key Performance Indicators

The identified Key Performance Indicators (KPIs) of the Vaccine Program (VP) and the Vaccine Management System (VMS) to measure success were as follows:

• 1.Support 100% of the Vaccine Administration Process at the NIH Main Campus and Remote Sites meeting all patient safety and mandatory reporting requirements. obtaining a <1% waste of the vaccine.
• 2.Provide a system that employees could schedule vaccinations and review the administration with more than a 90% satisfaction level.
• 3.Provide a system in which clinic staff have >90% satisfaction level of using the system.

Critical Success Factors

The identified critical success factors required to ensure success and the ability to meet the KPIs were as follows:

• 1.The need to align the business and technical aspects of the VP and VMS.
• 2.The need to ensure a culture of patient safety.
• 3.The software development life cycle (SDLC) to allow evolution and adaptability to address new and changes to requirements of the system in a timely manner.

Business and Technical Alignment

Leadership

The highest levels of the organization championed the vaccine clinic and were available to provide input and support for decisions that would affect the organization. At the operational level, the development of the vaccine clinic required levels of effort and careful coordination from leadership and staff across a large number of institutes, divisions, and departments at the NIH. Communication and involvement of leadership, accurate representation of the business needs, as well as input from stakeholders and staff who use the system are all critical to the success in the development of a software product.² The interdisciplinary team and various subgroups met twice weekly at the beginning of the project and when there were complex milestones to reach. The interdisciplinary team included members from IT, clinical informatics, health information management, medical, nursing, pharmacy, occupational health, finance, and safety/quality.

Clinic Description

The vaccination clinic was established in the hospital building's repurposed cafeteria, which remained unused during the period while the clinic was in operation. Physical distancing, enhanced ventilation, cleanable surfaces, and sufficient illumination were key to minimizing disease transmission and promoting worker safety during clinic hours. A review of the design of the clinic and the process is available at Golding et al.³ After vaccination, staff were given a completed vaccination card, a vaccine information sheet, and instructions to proceed to an observation area, and after the observation period staff exited through a separate door at the front of the cafeteria.

Pharmacy

The NIH Clinical Center's (NIH CC) pharmacy department managed COVID-19 vaccine inventory according to intricate and manufacturer-dependent storage and handling requirements. Pharmacists controlled the storage, thawing, preparation, and distribution of vaccine, in appropriate quantities, to ensure product stability and facilitate tracking of lot numbers and expiration dates.

Remote NIH Sites

The NIH National Institute of Allergy and Infectious Diseases Rocky Mountain Labs (NIAID-RML) and the National Institute of Environmental Health Sciences at Research Triangle Park, NC (NIEHS-NC) each had a vaccine clinic with similar set-up as NIH main campus that included check-in desk, area for administration of the vaccine, and observation area. The NIAID-RML clinic was staffed by volunteers from the NIH Office of Research Services and NIAID. The NIEHS-NC clinic was staffed by Occupational Medical Services and contract staff.

Culture of Patient Safety

A key to ensure patient safety in the implementation of new processes and systems is to identify real and potential risks early in the process. The NIH CC Office of Patient Safety and Clinical Quality facilitates a failure mode and effects analysis (FMEA) of the proposed process to review processes.

An FMEA allows an organization to review a process, the process steps, potential causes for a process to fail and the level of associated risk from high to low of processes that may fail. Those with moderate and high risks are reviewed with solutions to reduce the potential for failure and ensure patient quality and safety.⁴

FMEA sessions were activated to review processes associated with the administration of Moderna and Pfizer, and/or J&J during the same clinic days. The highest risk failure modes centered on staff communications. The full team met to review the process and findings of the FMEA. The outcome was the implementation of the following components before deployment:

• Enhancements to the VMS to ensure administration of the correct vaccine.

• Physical separation of staff and booths by manufacturer to ensure the correct vaccine would be administered.

• A card with the name of the manufacturer is handed to the patient to hand to the vaccinator at the booth, who verifies that the card matches the booth they are in.

• Clear signage to ensure staff being vaccinated reported to the correct booth.

Software Development Life Cycle

Before the pandemic the NIH CC followed a traditional SDLC with characteristics of Agile. There are multiple articles reviewing SDLC models identifying when it is appropriate to use each model.^5,6 The selected model depends on the organization, the time and level of risk to develop a minimal viable product, customer involvement, skills of the developers, size of the development team, characteristics of the organization and the end products. Reeves et al.⁷ identified the potential need to adapt an organization's formal SDLC under the time requirements and critical needs of the organization during a pandemic.

The Asymptomatic Staff Testing System (ASTS)⁸ developed during COVID-19, which had a 2-week development cycle from time of idea to deployment. ASTS provided a scheduling component, linkage to the NIH CC electronic health record (EHR) utilizing Allscripts Sunrise Clinical Manager referred to as the Clinical Research Informatic System (CRIS), an interface to the EHR for creating orders and populating medication administration data, and the Allscripts FollowMyHealth^® patient portal.

VMS was developed as a module to ASTS. During development of VMS, the organization had to be quicker, adapt continually, and include multiple workflows. Owing to these challenges, the SDLC utilized was a combination of the Spiral SDLC⁹ of repeatedly iterating a collection of development processes (i.e., define, design, develop, test, evaluate, and deploy) to reduce clinical risk and the Agile Methodology¹⁰ of delivering an incremental solution with evolving requirements and a strong focus on collaboration with stakeholders. This Spiral-Agile SDLC enabled the organization to develop a strong foundation of modules as part of the Spiral process and adapt the module through the Agile process.

To deliver vaccines to the community VMS deployment had a very short timeline of 2 weeks. The initial requirements included the following:

• 1.Allow the management of the list of staff to be vaccinated adding to the list and communicating to those added in a timely manner.
• 2.Allow the adaptation of the schedule over time.
• 3.Allow the ability to screen for and document any contraindications or precautions to vaccination.
• 4.Check-in staff to be vaccinated.
• 5.Manage both a two-dose and a single-dose regimen and all manufacturers.
• 6.Document the administration of the vaccine (all manufacturers).
• 7.Enable first-dose vaccinated staff to schedule a second dose.
• 8.Create the order and administration record within CRIS.
• 9.Create the immunization record in CRIS including availability in the patient portal.
• 10.Provide a vaccination card in pdf form in the patient portal.
• 11.Provide reports and dashboards to support process management for clinic management as well as for the pharmacy department.
• 12.Interface vaccine data to the appropriate state reporting system.
• 13.Provide real-time reports to clinic administration for ease in scheduling, planning, and reporting metrics to senior NIH leadership.

Since the vaccination program was an evolving initiative that continued to present new requirements, the system development had to continually adapt. The speed of modification and new requirement requests resulted in a scenario where developers were “building the plane while flying it,”⁹ a concept coined for iterative software development. This Spiral-Agile SDLC allowed the development of VMS to adapt to changing requirements with stakeholder input in a timely manner.

KPI Assessment

To continually evolve in implementing processes and systems it is critical to review the usage of a system, measurements of KPIs and review lessons learned from its use.

Vaccine Administration Process

From December 22, 2020 and June 10, 2021 there were ∼16,600 first doses and 15,800 s doses administered. For this period of time, 25 doses were wasted, which equals to 0.08%. The first two KPIs were met.

Vaccinated Employee Satisfaction

Between April 12, 2021 and May 22, 2021, a staff survey of individuals who received Dose 1 and/or Dose 2 was conducted with 2187 responses out of 5138 emails sent for a 42% response rate. The answers to the survey statements was based on a Likert scale, including Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree, and Not Applicable. Positive response rate was based on Strongly Agree and Agree and negative response rate was based on Disagree and Strongly Disagree (Table 1 and Figure 1).

Table 1.

Vaccinated staff survey questions

Q	Question
Q1	I understood the directions provided in the email that introduced this service.
Q2	It was easy to register for the vaccine using the website.
Q3	I was able to schedule my vaccine appointment without difficulty.
Q4	It was easy to register for the patient portal to view my vaccination record.
Q5	I understood the emails pertaining to scheduling my second dose of the vaccine.
Q6	In calling the patient portal support line, my questions were answered sufficiently.
Q7	The vaccination process started on time.
Q8	I had confidence and trust in the clinical staff who I encountered.
Q9	I had the appropriate level of privacy.
Q10	The provider explained what to expect before and after the injection in a way that I would understand.
Q11	All of my questions were answered to my satisfaction before getting the injection.
Q12	The staff working at the COVID vaccine clinic were friendly and professional.
Q13	It was easy to maintain 6 ft of social distance while in the observation area.
Q14	Staff utilized personal protective equipment such as gloves and masks in all areas of the vaccine clinic.

Figure 1.

Vaccinated staff survey results for positive and negative responses.

The results of the vaccinated staff survey showed that staff appreciated the opportunity to be vaccinated onsite and that the scheduling system was easy to use. From a systems perspective, the most challenges for staff was using the patient portal. A summary of the survey results is as follows. The comments collected through the survey demonstrated that 10% of respondents provided compliments regarding the clinic set-up/management, staff, and leadership. From a full clinic perspective, the greatest issue identified related to directions to the clinic and parking. The KPI addressing vaccinated staff satisfaction was met.

Clinic Staff Satisfaction

Staff who assisted with clinic operations who used the system as part of the check-in, administration of the vaccine, and/or management of the clinic were surveyed as well. Between April 12, 2021 and May 25, 2021, a survey of the employees who worked the NIH Covid-19 vaccine clinic was conducted with 60 responses out of 247 emails for a response rate of 24%. Positive and negative response rates followed the same calculation as the vaccinated staff survey (Table 2 and Figure 2).

Table 2.

Vaccine clinic staff survey questions

Q	Question
Q1	It was easy to check-in an employee.
Q2	It was easy to document the administration of the vaccine for the employee.
Q3	It was easy to complete the extra dose process.
Q4	The system was easy to use.
Q5	The scanner was easy to use.
Q6	The WoW was easy to use.
Q7	Training of the clinic processes was completed.
Q8	The conducted training allowed me to be successful.
Q9	The amount of time for the training of the vaccine check-in and administration application was adequate.
Q10	The content of the training of the vaccine check-in and administration application was adequate.
Q11	The amount of training of computer equipment including the WoW and scanner was adequate.
Q12	I used the dashboard views and reports to manage the clinic.
Q13	It was easy to navigate the dashboard views and reports.
Q14	The data on the dashboard views and reports were easy to follow.

WoW, Workstation on Wheels.

Figure 2.

Vaccine clinic staff survey results for positive and negative responses.

Functionality of the system met the needs of the staff who worked in and/or managed the clinic and it was easy to use. Staff reported that there was some difficulty with the scanner and with the Workstations on Wheels (WoWs). The primary hardware issue was ensuring the scanners and WoWs were fully charged. The primary usage issue with the WoWs involved the system timing out based on inactivity based on organizational security policy. This time out was extended to fit within security policy parameters, which improved user satisfaction of the system. The most commented item in the survey was that IT staff support were available in person during the first week of clinic operations and that they were very timely in responsiveness when needed throughout the operation of the clinic. The KPI addressing employee staff satisfaction was met.

Conclusion

In the case of a pandemic, organizations require the ability to develop successful systems in a timely manner. Using systems that are available, leveraging the Spiral-Agile hybrid SDLC to enhance systems, and engaging a collaborative interdisciplinary team and adding FMEA to continually review risk are key to developing an effective system to respond to organization needs during the pandemic.

This article reviewed the systems, roles, and SDLC needed to rapidly implement a successful organizational VP. The survey results of both those to be vaccinated and clinic staff showed the success of the systems, the workflow, and the clinics themselves.

Authors' Contributions

All authors wrote and reviewed the final article.

Authors' Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.