Skip to main content
. Author manuscript; available in PMC: 2023 Mar 1.
Published in final edited form as: Int J Med Inform. 2021 Nov 11;159:104643. doi: 10.1016/j.ijmedinf.2021.104643

Table 3: Illustrative Quotes and Technical Graphical User Interface Design Changes.

Vital Sign (VS), Graphical User Interface (GUI), Fraction of inspired oxygen (FiO2), Chronic Obstructive Pulmonary Disease (COPD), Medication Administration Record (MAR).

Theme Code Illustrative Quotes Technical Graphical User Interface Design Changes
1. Trend evolution CRI_RISK
BASELINE_RISK
PHYS_EXAM
PROX-REMO
HEMOD_HIGHTECH
HEMOD_LOWTECH
INTERVAL
OTHER_RISK
REPORT		 Participant MD1-3-2: “Respiratory wise, mostly it’s just visualization of work of breathing, FiO2 that’s being required with a pulse ox equivalent, and mostly looking at actual muscles of the chest wall moving…Then cardiovascular…warm touch perfusion extremities and then most of my patients will have a bedside ultrasound to give me a good visualization of…cardiac function.” Add:

  1. Scroll bar and zoom feature

  2. 3 and 4-day view options

  3. Up or down arrows next to risk score to emphasize score directionality.
Participant RN2-1-1: “…I’ll be honest it’s not popular with everybody, but bedside report can tell you a lot of things if you’re looking at the patient. It’s not just about the what’s on the monitor and…in a computer, it’s also looking at the patient as well, so tie all this together. Helps me…okay this patient is unstable. Keep a close eye on him, prioritize your patients, based on their acuity of issues.”
Participant RN1-1-2: “…I mean, this would be really great as like an adjunct to the nurse doing their nursing assessment and…trending your patients, critical thinking, having something like this, especially for our newer nurses.”
2. Context RISK_INDEX
BARRIERS
BENEFITS
PRACTICE		 Participant MD1-3-2: “…A concurrent FiO2 or…level of oxygen being applied to the patient would be helpful or…having some kind of…arrow on the graph where it says…6-liters, and then you give it 10-minutes, and all of a sudden you see 8-liters, then…15-liters, non-rebreather…It kind of shows you… that this is a respiratory failure…instead of something else. And…with an integration like another very…advanced thing is, I’d really like to see at least some drugs or the MAR…pop-up in this timeline.” Add:

  1. Ability to change the parameters of VS alerts for each VS, to control for individual differences (personalization of VS thresholds). **

  2. In proximity to risk index, indicate when interventions (FiO2, medications, etc.) were administered.
    1. Hover and discover function, use to prevent screen cluttering. **

  3. Power function to turn on/off for end-of-life scenarios.

  4. Do not add work- data must pull in automatically. **
Participant MD2-3-1: “We always under appreciate how much we harm people with medicine, and to have all the medications lined up versus what their vital signs are doing would really…show some of those risks.”
Participant MD2-2-1: “…One thing I really, really like about this…is that obviously these machine learning algorithms are simply a function of the input data and how often do we have pulse ox that are either erroneous or fall off? So, what I love about this is that if you see a change in the index, you can very quickly look to see if it’s actually clinically valid. For example, if the index goes sky high, saturations 55%, but at the same time you don’t have a confirmed tachycardia or tachypnea, it makes you sort of think maybe that was an erroneous value, so I love that it’s just not a black box number, but we actually have the data and the trends that we can refer back to.”
Participant MD2-2-1: “…I would just love to know which clinical features are really driving that index…It says saturation there, for example, but I think it’d be really great to know which actual feature is driving the change…”
Participant RN1-1-2: “…There’s always going to be those people that perhaps are already overwhelmed with the technology…So, they may not be as receptive to this because this is another task. Another thing that the nurses have to check at the bedside when they’re already doing a million things, but I think overall, the majority of people, as long as you didn’t ask them to do too much with it, would be happy to have another data point to look at.”
Participant RN2-1-4: “I start to hesitate whenever it adds additional steps for bedside staff. I get concerned for compliance and then does that impact the usefulness of the tool? There is a tendency to say oh I’ve got to do this? Well, I’m already doing X, Y and Z, so I’ll keep doing what I’m doing.”
Participant RN1-1-1: “…For example, somebody with COPD…They can bounce…between 88%…94%, but you have somebody else, like me, where you go down to 88% and we’re going to be struggling…Somebody that lives at certain ranges…does this account for any kind of inputs from us, and can we tailor it to certain patients?”
3. GUI evaluation / interpretation / explanation CHANGES
CONFIDEN
CRITIC_THINK
ALARM_FATIG
WEIGHT
ED
LIKE		 Participant RN1-1-4: “They may feel like…this doesn’t look that good, but I’m going to wait until it gets a little bit worse before I feel confident to call the physician. So, I think having this very tangible number that sort of justifies the phone call might make them feel confident to escalate earlier.”

  1. Remove VS name next to the risk index number.

  2. Brighten and enlarge the risk index column, include confidence intervals.

  3. Center the X and Y axes to normal (standardization of the axes).

  4. Desire to know clinical features driving the index score (add an information button that describes how the algorithm utilized VS features for index formulation).

  5. Add:
    1. Risk column and VS axes labels to left and right side of screen.
    2. Grid lines for finer unit of measurement.
    3. Hover and discover function to display pop-up boxes with VS values for an exact point in time. **
    4. Temperature and blood pressure to visualize on screen.
Participant RN1-1-1: “I’m going to be a pessimist on this. I think that new nurses need to learn how to critically think and if something like this is available to them, they’re not going to learn how to critically think…”
Participant RN1-1-2: “…. I do think there’s a lot of code shaming for people that when they call for help, you know, why did you call this and I think it gains confidence there. But it doesn’t take away the human aspect of having to physically assess your patient, having to look at them…You can’t go off from just numbers for a patient, you need to go and look at them.”
Participant MD2-2-1: “I guess one thing that I worry about, is that this as it is right now, I think puts us at risk of pretty significant alarm fatigue. And I think that could be alleviated by understanding what the y-axis…or even what the numbers mean in terms of score would be a little bit helpful.”
3a. Continuity of evaluation PRIORITIZE
HANDOFF		 Participant MD1-3-2: “[Electronic Health Record] is just awful at allowing me to trend vital signs and takes five minutes to load in three days’ worth of stuff. So, you know, that alone would be extremely beneficial, just to have an easier way to view.” There were no specific changes for this theme.
Participant RN2-3-4: “The physician pulled up a graph on the computer where it…trended the vital signs over a three-day period of time and then it was like blatantly obvious what was going on with the patient… It was really cool to see that…I think when you…take care of a patient for 12-hours and then pass to the next person, some of these very obvious trends can…get lost…as you pass the patient back and forth…”
4. Clinical intuition support GUT_FEELING Participant RN2-1-4: “…You know it’s kind of that gut feeling and we’ve looked at the trends…and think well, that may not be significant, kind of justify it in our heads…” There were no specific changes for this theme.
Participant RN2-1-1: “This can be helpful in a neuro environment or cardiac environment like [participant RN2-1-4] said. There are times, where you bet, you’ll think something’s wrong. But you don’t have enough data to back it up, it’s like okay let’s continue to monitor the patient, and then two to four hours later it’s worse. Well now, you…have data to back it up.”
Participant CRNP2-3-1: “The only thing that I would add is that, with all of that stuff that is objective there’s also just a gut feeling that you use a lot.”
5. Clinical operations utility EVIDENCE
STAFF		 Participant RN2-1-2: “…I’m a big fan of the monitors in the hallway…near the patient rooms, and I worry if it’s…in the nurse’s station that I’ll never see it. I don’t get to go in there very often on a busy day, and of course we have people who can come alert us, but it is more helpful if we can see it ourselves.”

  1. Add patient tabs across the top to toggle between patients. **

  2. Add and remove patients; customize assignment or role. **
Participant RN2-1-1: “…I know that clinicians and unit directors employ the use of dashboards so they may be able to look at this from a dashboard point of view…I know the charge nurses, clinicians, and unit managers really rely on the dashboards so that would be a place that it would need to be as well…”
Participant: MD2-2-1: “…When I’ve talked to some of the dinosaur attendings, particularly surgeons, they always worry about…replacing clinician gestalt…I think if there’s always that ability to clinically validate the scores that are generated, I think that would really, really help with buy in…”
Participant RN2-1-4: “I have interest in this from a staffing and assignment perspective…this would be a helpful tool…I know that unit staffing doesn’t always mirror actual patient need as much as we would like it to, but this could give very real data to show a population on a certain unit, their patients are always running in the eights and nines. Maybe that gives leadership some data to say we need more or we need X, Y, or Z because our patients, not only do we feel they have a higher acuity we can prove it. And then, also for assignment making you know, do we, you know give nurses higher acuity patients and less of them, do we dispense…the number of patients on the floor with high acuity and mix those with low…But it does give us very real numbers to kind of make some of those decisions that otherwise you know we do on the day to day.”
Participant MD2-3-1: “When it’s implemented, I really wouldn’t put any real faith into it right away. You know roll it out slowly, with some good kind of education and take some time to catch on. But I think you know all around the country people have been trying to make these early warning systems and they’ve all been fraught with different problems…making them clinically difficult to implement…”

Most GUI changes can be visualized in Figures 2, 3a, and 3b. Some changes are interactive and would not be appreciated in a static figure. These changes were applied to the final GUI prototype in the interactive development software and are denoted as (**).

The table above provides participant quotes that were coalesced between both rounds. Quotes were analyzed to discover the qualitatively reported themes.