Skip to main content
PMC home page
Health Systems logoLink to Health Systems
. 2019 May 28;8(3):155–161. doi: 10.1080/20476965.2019.1620638

Development of a computerized paediatric intensive care unit septic shock pathway: improving user experience

Lauren Shivers a, Sue S Feldman b, Leslie W Hayes c,✉
PMCID: PMC6896506  PMID: 31839928

ABSTRACT

The purpose of this study was to understand the user experience with a computerized septic shock protocol relative to the workflow of Paediatric Intensive Care Unit clinicians. The need for data-driven, condition-specific, computerized protocols in the intensive care unit helps improve decision-making at the bedside. PICU clinicians were interviewed and given pre-and post-implementation surveys asking their opinions on the current PICU septic shock protocol, as well as the current electronic health record being used at [Paediatric Academic Medical Center]. User preferences guided adjustments toward improved usability of the septic shock protocol. Computerized Physician Order Entry, a critical component of the septic shock protocol, allows for more streamlined processes, more complete records, and more time to care for patients. This study revealed that although clinicians had an unfavorable view of the EHR in general, the computerized septic shock protocol was very well-received with an overall usability score of 82.

KEYWORDS: quality improvement, health informatics, user experience, clinical protocols, electronic health records, order entry systems

1. Introduction

In 2005, the World Health Organization (WHO) reported that sepsis accounts for 60–80% of lost lives per year in childhood (Kissoon et al., 2011, p. 494). The World Federation of Paediatric Intensive Care and Critical Care Societies (WFPICCS) announced in 2007 the launch of a quality improvement initiative called The Global Paediatric Sepsis Initiative, to provide guidance for care providers in recognizing and treating sepsis and septic shock in paediatric patients (Kissoon et al., 2011). Studies have revealed the importance and success of early recognition and treatment of septic shock in paediatric patients (Kissoon et al., 2011; Rivers et al., 2001). Guidelines for management and treatment of sepsis and septic shock are regularly updated (Brierley et al., 2009; Dellinger et al., 2013). However, the literature on the efficiency of computerized septic shock protocols (SSP)s as a more standardized process for bedside clinicians is very limited.

Studies have also revealed the favourable attitude of bedside clinicians with computer experience, towards documentation in the electronic health record (EHR) and the ability to standardize care and improve patient outcomes (Middleton et al., 2013; Paul et al., 2014). As technology improves, this research is also evolving, along with the changing demographic of bedside clinicians, such as the movement from non-digital natives to digital natives (Palfrey & Gasser, 2011). Computerized protocols used by varying clinicians have been shown to decrease clinical errors and improve decision-making at the point of care (Morris, 2001). The need for data-driven, condition-specific, computerized protocols in the intensive care unit (ICU) has been shown to individualize patient care, while also standardizing processes and decision-making at the bedside for clinicians (Morris, 2001).

The user experience (UX) research relative to the implementation of paediatric computerized SSPs is very limited. Because the users are a changing demographic (i.e. medical residents and fellows are digital natives) (Palfrey & Gasser, 2011), it is vital to tailor the visualization of information to best meet their needs, especially the information being used at the point of care (Press et al., 2015). This study examined the paper process and sought to understand the user experience of a computerized SSP relative to the workflow of PICU clinicians and the standardization of care at [Paediatric Academic Medical Center] (PAMC). PAMC is the only free-standing, tertiary care paediatric hospital in the state of [state]. The PICU contains 24 patient beds and receives over 1,500 admissions per year. Because of the need to quickly assess and treat paediatric septic shock patients, coupled with the fast-paced PICU environment, researchers recognized the need for a computerized, condition-specific clinical pathway in the EHR to streamline septic shock care and standardize processes at the bedside.

There are a variety of clinical care environments in which septic shock is treated. The literature is consistent in the fact that the sooner in the septic cycle that care is rendered

and the more consistent that care, the better the outcomes (Rivers et al., 2001). Complicating optimal delivery of this care is the complex environment in which these patients present: emergency departments, burn units, ICUs, etc. (Cruz et al., 2011). Because these environments each have multiple vectors from which complex pieces of data flow at fast rates to influence decision-making, computerized protocols balance the complexity and rate with the limits of human decision-making (Press et al., 2015).

The Paediatric Intensive Care Unit (PICU) SSP at PAMC was previously in paper form and utilized by nurses, mid-level providers (Certified Registered Nurse Practitioners, CRNP), and physicians. This SSP was developed based on best evidenced-based practice, incorporating the most recent paediatric septic shock guidelines, implemented with consideration for the local context (Brierley et al., 2009). A computerized form of the pathway incorporated into the current EHR could standardize and improve workflow and decision-making at the bedside for PICU clinicians, thereby optimizing care, embedding improvement into daily work, and potentially decreasing errors at the point of care.

2. Materials and methods

Because the PICU SSP initially started on paper and was then implemented as a computerized protocol in the PAMC EHR, it was necessary to perform both formative and summative user evaluation (Currie, 2005). Formative user evaluation was conducted during the pre-implementation phase in January 2018, and summative user evaluation was conducted during the post-implementation phase in August 2018. The purpose of the pre-implementation phase was both to inform the design of the new workflow for computerized SSP as well as to improve the implementation process of the workflow. Data collection was accomplished by surveys during the pre-implementation phase (January 2018) and use of the System Usability Scale (SUS) during the post-implementation phase (August 2018). The SUS is an appropriate instrument for post-implementation evaluation because it is a reliable instrument that allows for assessing subjective responses and can be accomplished with few participants (Tullis & Stetson, 2004). A sample of PAMC PICU bedside clinicians, broadly representing all bedside providers (i.e. Registered Nurses, Advanced Practice Nurses, such as Nurse Practitioners, and physicians) were recruited in person during staff meetings held in the PICU.

Paper survey administration included the survey and a designated, pre-labelled envelope for survey return. Of the 20 surveys distributed, 10 were returned. All 10 surveys were usable for analysis. SUS literature suggests that the ideal sample size for use of the SUS instrument is 12, citing that 12 surveys can yield 90% of the accurateness when compared to a sample size of over 100 (Tullis & Stetson, 2004). Participants’ names were not included on the surveys as to protect anonymity. This study was conducted under the University of Alabama at Birmingham Institutional Review Board # 300,001,009.

2.1. Computerized SSP development

Development, deployment, and evaluation of the computerized SSP was guided by industry best practices as outlined in the software development life cycle (SDLC) (Wager, Lee, & Glaser, 2017). This was accomplished during a think-aloud session with PICU clinicians and two nurse informaticists. Feedback was recorded by the researcher and nurse informaticists. Suggested changes were compared between the researchers and the nurse informaticists and discrepancies were discussed and resolved with the focus on improving usability. The design of the computerized SSP incorporated changes suggested during the think-aloud session as well as general concerns gathered during the pre-implementation survey. Next, testing of the computerized SSP began per the Nursing Informatics department at PAMC. Following the testing and implementation of the computerized PICU SSP, users were asked to complete the SUS instrument. The requirements gathering and implementation pathway for the computerized SSP is illustrated in Figure 1.

Figure 1.

Figure 1.

Open in a new tab

Requirements gathering and implementation pathway for computerized SSP.

2.2. The system usability scale methodology

The SUS developed by John Brooke (Brooke, 1996) was utilized in this study as a lens through which to understand the overall user experience of the computerized SSP. These measures of usability encompass users’ effectiveness (users’ ability to complete tasks), efficiency (how well and quickly users can complete the task), and satisfaction (users’ reaction to completing the tasks) (Brooke, 1996). The 5-point Likert scale allows users to choose responses ranging from strongly disagree (1) to strongly agree (5). See Table 1 for the SUS as adapted for this study.

Table 1.

System Usability Scale (Adapted from (Brooke, 1996)).

  Strongly Disagree       Strongly
Agree
1. I think that I would like to use this Septic Shock Protocol frequently. ◯ ◯ ◯ ◯ ◯
2. I found the Septic Shock Protocol unnecessarily complex. ◯ ◯ ◯ ◯ ◯
3. I thought the Septic Shock Protocol was easy to use. ◯ ◯ ◯ ◯ ◯
4. I think that I would need the support of a technical person to be able to use this Septic Shock Protocol. ◯ ◯ ◯ ◯ ◯
5. I found the various functions in this Septic Shock Protocol were well integrated. ◯ ◯ ◯ ◯ ◯
6. I thought there was too much inconsistency in this Septic Shock Protocol. ◯ ◯ ◯ ◯ ◯
7. I would imagine that most people would learn to use this Septic Shock Protocol very quickly. ◯ ◯ ◯ ◯ ◯
8. I found the Septic Shock Protocol very cumbersome to use. ◯ ◯ ◯ ◯ ◯
9. I felt very confident using the Septic Shock Protocol. ◯ ◯ ◯ ◯ ◯
10. I needed to learn a lot of things before I could get going with this Septic Shock Protocol. ◯ ◯ ◯ ◯ ◯
Open in a new tab

The SUS scores range from 0 to 100, with the participant’s scores for each question added, then multiplied by 2.5 to give a final score (System Usability Scale, n.d.). Research has shown that scores at or above 68 are considered above average usability, with scores below 68 considered below average usability (Bangor, Kortum, & Miller, 2008; Brooke, 1996). These final SUS scores provide a reliable evaluation tool to measure usability (Brooke, 1996). Paper surveys were used instead of electronic surveys due to clinicians using varying computers in various locations, as well as the inability to upload surveys on hospital computers.

3. Results

As previously mentioned, of the 20 surveys distributed, 10 were returned yielding a 50% response rate with the following breakdown: bedside RNs n = 5, prescribing clinicians (Advance Practice RN and physicians) n = 5. This sample represented 100% of the Advance Practice RNs and 30% of the critical care physicians employed at the PAMC PICU at the time. All respondents had only ever been employed at PAMC and used its EHR (AllScripts©). Of the physicians, 66% (n = 3) had used EHRs in addition to that used at PAMC. Of the nurse respondents (n = 5), PICU job experience ranged from 1 to 34 years and all had only ever used AllScripts©. Figure 2 illustrates further distribution in terms of years at PAMC and year in the PICU.

Figure 2.

Figure 2.

Open in a new tab

Years in PICU, Years at PAMC. Years using Allscripts by Respondent.

The findings below are organized in the order in which the study was conducted: pre-implementation survey, think aloud session, post-implementation survey.

3.1. Pre-implementation survey results

The purpose of the pre-implementation survey was to assess the clinicians’ perceptions of the EHR used at PAMC, along with any barriers to its use. In this baseline survey, 100% of physicians felt the EHR improved the quality of care of their patients, while only 60% of nurses felt it improved care. Similarly, all physicians felt confident documenting in the EHR while caring for septic shock patients, while 80% of nurses and none (0%) of the Advance Practice Nurses felt confident. In this pre-implementation phase, we also asked about how confident clinicians were that they could find all of the SSP orders within the EHR (which are from the paper SSP). Advance Practice Nurses felt most confident (100%, n = 2), with Registered Nurses next most confident 80% (n = 4) and physicians were least confident that they could find all of the SSP orders in the EHR 66% (n = 2). Other findings include 20% (n = 1) of prescribing clinicians (physicians and Advance Practice Nurses), and none (0%) of nurses said the orders for the SSP from the paper protocol were easily documented in the EHR. All physicians and Advanced Practice Nurses (100%, n = 5) and 80% (n = 4) of nurses felt all information needed to carry out the SSP was available in the EHR. Finally, 100% of physicians (n = 3), 80% of nurses (4), and zero (0%) Advance Practice Nurses felt that all necessary information needed to complete an appropriate order set was available in the EHR. There was uniform belief among those surveyed that a computerized SSP in the EHR could improve patient quality of care by allowing septic shock orders to be quickly located in a more consolidated location, with 100% of prescribing PICU clinicians (n = 5), and 80% (n = 4) of nurses holding this opinion (See Table 2.)

Table 2.

Stratified pre-implementation survey findings by question; results represented as means for each group.

  Physicians (n = 3) Nurses
(n = 5)		 Advance Practice Nurse
(n = 2)		 Overall
(n = 10)
EHR improved quality of care for patients 100% 60% 0% 60%
I feel confident in documenting in the EHR 100% 80% 0% 70%
Ease of SSP order location 66% 80% 100% 79.8%
Felt SSP order well documented in the EHR 20% 20% 0% 16%
Felt all info needed for SSP available in the EHR 100% 100% 80% 96%
Felt all info needed to complete order set 100% 80% 0% 70%
Computerized SSP could improve quality of care 100% 80% 100% 90%
Open in a new tab

3.2. Think aloud session

The purpose of the think-aloud session was to collect data at the point of care while prescribing PICU clinicians (physicians and advanced practice nurses) performed tasks in the EHR relative to the septic shock order set. Findings from the think-aloud session were incorporated into the development modifications to the computerized SSP. Such modifications from the think-aloud session included:

  • changing a drop down menu from “inpatient vs observation” to “inpatient” alone,

  • changing neurological assessment from every 4 hours to every hour,

  • adding CRNP to notification choices (choices were initially only physicians) (CRNP is an advance practice nurse, Certified Registered Nurse Practitioner),

  • adding vital sign parameters for notifications based on age,

  • changing daily chest x-ray to 10 PM for 7 days,

  • changing the ionized calcium order from hourly to hourly as needed,

  • allowing strict intake and output to be an automatically selected order, and

  • allowing the near-infrared spectroscopy to be an automatically selected order.

3.3. Post-implementation survey results

The purpose of the post-implementation surveys was to assess the usability of the implemented computerized SSP. As mentioned previously in this paper, the SUS was employed as a means to measure SSP usability. Findings revealed that the average usability score for the computerized SSP was 82 with equal distribution among Registered Nurses, Advanced Practice Nurses, and Physicians (see Figure 3). In terms of looking at individual questions, the SUS survey found that 100% of nurses (n = 5), 100% of advanced practice nurses (n = 2), and 100% of physicians (n = 3) would like to frequently use the computerized SSP. All of those surveyed (100%, n = 10) felt confident using the computerized SSP, and none of the respondents felt the need for a technical person to assist in the use of the computerized SSP. Likewise, all of those surveyed (100%, n = 10) found the various functions of the computerized SSP in the EHR to be well integrated. Additionally, findings showed that none of the clinicians surveyed found the computerized SSP in the EHR cumbersome to use.

Figure 3.

Figure 3.

Open in a new tab

Pareto Chart demonstrating pre- and post-implementation survey response rates as stratified by roles.

Figure 4 represents a Pareto chart that demonstrates the overall distribution of the SUS survey score ranges for all ten responses. These results reveal an overall highly favourable usability of this newly implemented computerized SSP.

Figure 4.

Figure 4.

Open in a new tab

Pareto Chart demonstrating post-implementation SUS survey score ranges.

4. Discussion

In terms of the EHR at PAMC, our findings align with that in the literature in suggesting that clinicians often see the EHR as a barrier to time efficiency; too many clicks while using the EHR; and having multiple software issues (Moody, Slocumb, Berg, & Jackson, 2004). This study also corresponded with the literature by demonstrating bedside clinicians holding favourable attitudes toward the idea of a computerized SSP in the EHR because of the ability to standardize care and have a consolidated location for septic shock orders and increased speed due to the availability of a specific septic shock order set (Morris, 2001). Post-implementation results suggested a favourable attitude with regard to the computerized SSP in the EHR and the ability to improve decision-making at the point of care. The findings in this study also displayed the importance of end-user input to assist in determining needs for improved functionality in the EHR. From previous anecdotal experiences, the researchers in this study expected to find the users’ frustrations with the EHR; however, researchers were pleased to discover the favourable attitude towards the PICU SSP in the EHR, even with the clinicians’ frustrations with basic EHR functionality. This finding is particularly significant given the baseline unfavourable attitude toward the ability of this PAMC EHR to assist in providing effective, efficient care.

The results of this study were reported back to PICU staff during a PICU Quality Improvement meeting and shared broadly with all PICU staff. Results were also shared with the Nursing Informatics department to assist in any future changes to the computerized SSP based on user preferences, as well as an example of optimal migration from a paper process to an electronic version at PAMC. If any future changes are made to the computerized SSP, a follow up study could be planned to assess usability after those modifications.

It is important to note that with the successful implementation of this SSP in the EHR, PICU nurses partnered with the PAMC Simulation Centre to develop a paediatric critical care nurse-focused simulation curriculum to improve nursing knowledge of disease-specific pathways, in this case, septic shock, and their management. A scenario using the computerized SSP is taught every month in the simulation centre for 8 PICU nurses at a time. As part of another, yet related, initiative, so far 40 nurses have completed this course with improvement in overall confidence and satisfaction with knowledge and skill using the electronic SSP and caring for critically ill children with sepsis and septic shock.

The PAMC PICU Quality Improvement council continues to monitor and analyse both process and outcome measures from the implementation of the computerized SSP. If gaps in performance are identified, this council will respond with improvements as needed. The computerized SSP has not only improved ease of implementation and documentation of key elements of care in critically ill children with septic shock, it is a process measure for documentation of a team huddle and debrief to assess adherence to SSP bundle elements.

Findings from this study also led to future changes to the review process for EHR order sets and clinical pathways at PAMC. By involving the users and their feedback, researchers set precedence for an interprofessional approach to implementing order sets and clinical pathways in the EHR. Due to its success, this study became a model for future order set and clinical pathway implementations at PAMC.

4.1. Limitations

This study had several limitations that may limit its generalizability. First, the sample size was fewer than the recommended number. Ideally, the study would have included 24 participants (12 prescribing (i.e. physicians and advance practice nurses) and 12 non-prescribing (i.e. RNs). Doing so would have strengthened the results of this study and added to generalizability. This study was conducted during a period of record high acuity as well as census in the PAMC PICU; the clear need to proceed with implementation of this computerized SSP contributed to smaller sample size. Also, during the think-aloud session, the PICU Attending Physician was present while the study participants interacted with the computerized SSP and gave feedback accordingly. The presence of the Attending Physician could have injected bias in the study because participants could have changed their opinions and/or behaviours while being observed.

5. Conclusion

This study reported on the findings of the process of converting a paper process to a computerized process, in this case, a septic shock protocol. The findings suggest that involving users early in the development and then revisiting with the users post-implementation is an effective manner in which to implement new processes in high stress, high volume, high stakes environments such as a PICU. We believe this has broad implications for a more collaborative and user-centered design approach to future clinical pathways at PAMC.

Acknowledgments

The authors thank the COA PICU staff for their willingness to participate in this study as well as the COA nurse informatics team for their collaboration on development of the computerized SSP. We wish to acknowledge the UAB School of Health Professions MSHI program for encouraging and supporting this work. Lastly, thank you to the reviewers whose review of this manuscript helped to make it a much better paper.

Disclosure statement

No potential conflict of interest was reported by the authors.

References

  1. Bangor A., Kortum P. T., & Miller J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. [Google Scholar]
  2. Brierley J., Carcillo J. A., Choong K., Cornell T., DeCaen A., Deymann A., … Dugas M.-A. (2009). Clinical practice parameters for hemodynamic support of pediatric and neonatal septic shock: 2007 update from the American College of Critical Care Medicine. Critical Care Medicine, 37(2), 666. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Brooke J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4–7. [Google Scholar]
  4. Cruz A. T., Perry A. M., Williams E. A., Graf J. M., Wuestner E. R., & Patel B. (2011). Implementation of goal-directed therapy for children with suspected sepsis in the emergency department. Pediatrics, 127(3), 2010–2895. peds. [DOI] [PubMed] [Google Scholar]
  5. Currie L. M. (2005). Evaluation frameworks for nursing informatics. International Journal of Medical Informatics, 74(11–12), 908–916. [DOI] [PubMed] [Google Scholar]
  6. Dellinger R. P., Levy M. M., Rhodes A., Annane D., Gerlach H., Opal S. M., … Jaeschke R. (2013). Surviving Sepsis Campaign: International guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Medicine, 39(2), 165–228. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Kissoon N., Carcillo J. A., Espinosa V., Argent A., Devictor D., Madden M., … Latour J. (2011). World federation of pediatric intensive care and critical care societies: Global sepsis initiative. Pediatric Critical Care Medicine, 12(5), 494–503. [DOI] [PubMed] [Google Scholar]
  8. Middleton B., Bloomrosen M., Dente M. A., Hashmat B., Koppel R., Overhage J. M., … Zhang J. (2013). Enhancing patient safety and quality of care by improving the usability of electronic health record systems: Recommendations from AMIA. Journal of the American Medical Informatics Association, 20(e1), e2–e8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Moody L. E., Slocumb E., Berg B., & Jackson D. (2004). Electronic health records documentation in nursing: Nurses’ perceptions, attitudes, and preferences. CIN: Computers, Informatics, Nursing, 22(6), 337–344. [DOI] [PubMed] [Google Scholar]
  10. Morris A. H. (2001). Rational use of computerized protocols in the intensive care unit. Critical Care, 5(5), 249. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Palfrey J. G., & Gasser U. (2011). Born digital: Understanding the first generation of digital natives: ReadHowYouWant. com,Basic Books, Philadelphia, PA. [Google Scholar]
  12. Paul R., Melendez E., Stack A., Capraro A., Monuteaux M., & Neuman M. I. (2014). Improving adherence to PALS septic shock guidelines. Pediatrics, 133(5), 2013–3871. peds. [DOI] [PubMed] [Google Scholar]
  13. Press A., McCullagh L., Khan S., Schachter A., Pardo S., & McGinn T. (2015). Usability testing of a complex clinical decision support tool in the emergency department: Lessons learned. JMIR Human Factors, 2(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Rivers E., Nguyen B., Havstad S., Ressler J., Muzzin A., Knoblich B., … Tomlanovich M. (2001). Early goal-directed therapy in the treatment of severe sepsis and septic shock. New England Journal of Medicine, 345(19), 1368–1377. [DOI] [PubMed] [Google Scholar]
  15. Tullis T. S., & Stetson J. N. (2004). A comparison of questionnaires for assessing website usability Usability professional association conference (Vol. 1), Minneapolis. [Google Scholar]
  16. Wager K. A., Lee F. W., & Glaser J. P. (2017). Health care information systems: A practical approach for health care management. John Wiley & Sons, Josey and Bass. [Google Scholar]

Articles from Health Systems are provided here courtesy of Taylor & Francis

RESOURCES