“A catalyst for action”: Factors for implementing clinical risk prediction models of infection in home care settings

Abstract

Objective

The study sought to outline how a clinical risk prediction model for identifying patients at risk of infection is perceived by home care nurses, and to inform how the output of the model could be integrated into a clinical workflow.

Materials and Methods

This was a qualitative study using semi-structured interviews with 50 home care nurses. Interviews explored nurses’ perceptions of clinical risk prediction models, their experiences using them in practice, and what elements are important for the implementation of a clinical risk prediction model focusing on infection. Interviews were audio-taped and transcribed, with data evaluated using thematic analysis.

Results

Two themes were derived from the data: (1) informing nursing practice, which outlined how a clinical risk prediction model could inform nurse clinical judgment and be used to modify their care plan interventions, and (2) operationalizing the score, which summarized how the clinical risk prediction model could be incorporated in home care settings.

Discussion

The findings indicate that home care nurses would find a clinical risk prediction model for infection useful, as long as it provided both context around the reasons why a patient was deemed to be at high risk and provided some guidance for action.

Conclusions

It is important to evaluate the potential feasibility and acceptability of a clinical risk prediction model, to inform the intervention design and implementation strategy. The results of this study can provide guidance for the development of the clinical risk prediction tool as an intervention for integration in home care settings.

INTRODUCTION

Home care, in which patients receive health and care services by a range of professionals, predominantly nurses, is increasing internationally.¹ For example, in the United States, nearly 4.5 million individuals received care from over 12 200 home care agencies in 2015, with overall spending on home health reaching $102.2 billion in 2018.^2,3 Healthcare-acquired infections are an important issue for all healthcare environments including home care,⁴ in which it has been identified that up to 17% of all unplanned admissions to hospitals from home care environments in the United States were due to infection.⁵ In a recent study, we developed a clinical risk prediction model using routine clinical assessment data collected during a patient home care episode to predict a patient’s risk of infection-related hospitalization or emergency care.⁶ The model identified over 30 factors for infection risk in home care patients and provided cutoff points or thresholds for determining which patients have a high or very high risk.⁶ Predictions from this model could be used by home care agencies to target resources more efficiently, and by home care nurses to target their plan of care for patients to maximize infection prevention.

The clinical risk prediction model we have developed is an example of a clinical prediction rule. Clinical prediction rules combine multiple pieces of information such as patient characteristics or test results (predictors) to estimate a patient’s probability of having either a current condition (diagnostic model) or a future health condition (prognostic model), using statistical techniques.^7–9 A significant number of clinical prediction rules or models have been developed in health and psychological sciences. A meta-analysis of 136 studies comparing statistical prediction with clinical judgment found that statistical and prediction rules outperformed clinicians in up to 47% of studies and that clinical judgment only outperformed prediction models in 6%-16% of the studies analyzed.¹⁰ One of the reasons why there has been interest in the use of clinical prediction rules in clinical settings is that they are perceived to overcome some of the failings or flaws in human reasoning. Clinicians using experience or expertise to inform their diagnoses or predictions are at risk of using subconscious heuristics or bias, which may lead to significant errors in judgment.^11,12 Using the dual process theory of reasoning, in which system 1 (S1) reasoning (subconscious, intuitive, context-based) is thought to lead to errors and S2 reasoning is the more analytical, conscious reasoning process¹²; clinical prediction rules are theorized to provide a S2 oversight for S1 thinking to try and reduce cognitive errors and bias.

Despite the increasing publication of clinical prediction rules, few studies have explored their clinical impact.^8,13 Clinical prediction rules are a form of clinical decision support. Clinical decision support systems are “designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient-specific assessments or recommendations that are then presented to clinicians for consideration.”¹⁴ Clinical prediction rules are often implemented with the assumption that their use will alter clinician judgment (in our case the identification of a patient at risk of infection that would otherwise not be identified), impact decision making and actions taken to mediate the infection risk, and prevent an infection from occurring.¹⁵ Assessing the impact of such interventions is complex; it is often not enough to provide clinicians with the output or risk estimation from a clinical risk prediction model to change behavior. In a study exploring the use of a predictive risk model for postoperative nausea and vomiting. Kappen et al⁷ highlighted that not providing treatment recommendations alongside the model output increased the clinicians’ cognitive burden (as they were having to translate the probabilistic information into actions and management strategies themselves). A number of studies have highlighted that to be effective, decision support interventions (eg, clinical prediction models) need to have a clear advantage over regular ways of working, and the tools need to be credible, should be integrated into clinician workflow, should be available at the point of decision making, and should be easy to use.^16–19

In order to ensure a clinical risk prediction model is integrated into practice, a number of issues need to be addressed, including acceptability to clinicians and how the tool can be provided at the point of care.⁹ Wallace et al⁹ outlined four phases of applying a clinical risk prediction model in clinical practice (Figure 1). We have completed the exploratory phase (I) by developing and testing the clinical risk prediction model.⁶ In this article, we report on qualitative data to inform the preparation phase (II) in a clinical setting. The aim of this article is to outline how a clinical risk prediction model for identifying patients at risk of infection is perceived by home care nurses, and to inform how the output of the model could be integrated into clinical workflow. This will then be used to inform our intervention development.

Figure 1.

Phases for impact analysis of clinical predication risk models. Adapted with permission from Wallace et al.⁹

MATERIALS AND METHODS

The qualitative methods used in this study have been fully described elsewhere.¹⁵ Semi-structured interviews were conducted with home care nurses to explore nurses’ judgment and decision processes surrounding infection risk and infection control practices (results published elsewhere).¹⁵

Setting

The study setting was a large Medicare-certified not-for-profit home care agency located in the Northeast region of the United States. In 2019, it provided over 1 million professional clinical visits and employed nearly 1500 nurses. When a patient is admitted to the agency, a standardized assessment, known as the Outcome and Assessment Information Set (OASIS)²⁰ is conducted by the home care nurse. Data from this assessment are used by the agency to calculate risk prediction scores, such as a hospitalization risk score,²¹ which can then be used by home care nurses to inform interventions such as frontloading care visits.²²

Nurse recruitment

Nurses were recruited to the study using a mixture of purposive and snowball sampling. Recruitment methods included email invitations to participate in the study and through researchers’ attendance at staff meetings. All nurses providing care in patients’ homes were eligible to participate. A total of 50 nurses participated in the study and were interviewed, with a mean age of 47 (range, 25-69) years. Participants had worked in a home care setting for an average of 13 (range, 3 months to 26 years) years and had an average of 19 (range, 1.5 to 44) years of nursing experience. The participants were also racially and ethnically diverse, with nearly half the sample (n = 24, 48%) identifying as Black/African American and 10% identifying as Hispanic.

Data collection

Interview data were collected from December 2017 to September 2018. Interviews were conducted by 2 research fellows, both trained in qualitative research. The majority of the interviews were conducted over the telephone, with an average time of 31 (range 18-50) minutes. All interviews were audio-recorded and then transcribed before analysis. All nurse participants were offered a $100 gift card for their time completing study activities. As highlighted previously, as part of the interview nurses, were asked about their knowledge and understanding of clinical prediction rules, and if or how a tool to help them identify a patient’s risk of infection could be used in practice. In addition to the questions listed in Dowding et al,¹⁵ at the end of the interview we asked the nurses specific questions related to their perceptions of potential usefulness of a prediction score as follows:

1. Do you think that targeted information about a patient’s risk for developing an infection would help you develop the plan of care? If so, how?

2. Do you think you would change what you did if you knew ahead of time that a patient was at an increased risk for developing an infection? If so, how?

3. Do you think that an infection risk score would be useful?

Analysis

Interview transcripts were imported into qualitative analysis software (NVivo Version 11; QSR International, Melbourne, Australia). The analysis was undertaken by 3 members of the research team (D.D., D.R., M.T.) who independently identified themes for an initial coding scheme. The coding schemes were synthesized into an overarching coding scheme, which was used to code interviews. Interrater reliability coefficients were calculated (k = 0.57; 95.9% agreement) to identify areas of agreement and disagreement, followed by modification of the coding scheme. Coding of the interviews and modification of the thematic framework were undertaken iteratively to produce a final coding scheme that was then applied to all 50 interview transcripts. All transcripts were coded by 2 researchers (final agreement: k = 0.73; 92.4% agreement), before a synthesis of key themes and issues arising from the data was produced.

Ethics

Informed written consent was obtained from each nurse before the interview. Their participation was voluntary and they could withdraw from the study at any time. The study was approved by home care agency’s institutional review board (IRB) (IRB#:E16-005) and the IRB for the collaborating researchers’ institution (IRB#:AAAQ9226).

RESULTS

Two main themes were derived from the data related to clinical risk prediction models for infection. These themes included (1) informing nursing practice, which outlined how a clinical risk prediction model could inform nurse clinical judgment about a patient’s level of risk for infection and be used to modify their care plan and interventions, and (2) operationalizing the score, which summarized nurses’ thoughts about how the clinical risk prediction model could be incorporated within the home care setting. Each theme is discussed in turn, with quotation numbers (Q#) referred to in the text and individual nurse responses identified by an ID number.

Theme 1: Informing nursing practice

Many of the nurses discussed how a score or prediction for infection risk would be useful or helpful for their practice, in terms of providing more information about the patients and helping them to focus on certain aspects of a patient’s condition or home environment (Table 1). They discussed how it would help them to identify and focus specifically on a patient’s risk, alerting them to factors that they otherwise may not consider (quote 1 [Q1] to Q2). Some of the nurses highlighted how the clinical risk prediction model could be useful by confirming what they suspected about a patient’s level of risk for infection, making their judgment visible to others, rather than having a utility for informing their own practice (Q3). Or the model could act as a second opinion, to make sure they did not miss anything (Q4).

Table 1.

Informing nursing practice: Useful for practice

Identify specific risk Quote[Q]1: “Anything that could kind of give us a little bit more information. Because like I said, now we really ‘have to be detectives now.’” (ID217) Q2: “That’s very helpful because not everything you’re going to know what to look for. Because if you have that, you’ll know exactly what to look for.” (ID218) Use of the prediction risk score

Q3: “If someone has a higher level, I think we're really going to tune in to that. Not that we wouldn't anyway, but I think it's like I said, like a reinforcement, a reminder of looking at that and considering all the different possible contributing factors.” (ID211)

Q4: “But if there was something that could signal to me, I mean, ‘Be careful. This patient is at high risk of infection,’ it could serve as a second opinion to me.” (ID213) Strategies for infection prevention Q5: “So I guess if somebody’s high risk of something, then you might incorporate something a little more in the treatment plan. Right. And maybe approach it a little different.”(ID620) Q6: “And this way patients that are at higher risk would be more monitored for infection, although we always monitor every patient for infection. But if they're higher risk, we probably would be more attentive to them and their diseases, whatever the risk factors are.”(ID760) Teaching interventions Q7: “If a patient has this type of wound, and the risk of developing infection is this percentage, we will focus more on the teaching of how to prevent in order to decrease that percentage of risk of infection.” (ID210) Q8: “Absolutely, because then you can, like I said, you can do things differently. You can educate the patient, and you can educate the family. You can, basically, try to prevent the patient from developing an infection if they're at a high risk.”(ID310) Follow infection control protocols more closely Q9: “You would be more diligent. You would be more follow the protocols.”(ID212) Q10: “So I guess if you can show us that, ‘Oh, you know what? This guy's scores 95% on the infection,’ there's a very high risk of infection, then we would be probably more likely to be more cognizant of hand hygiene and things like that.” (ID214) Effective implementation Q11: “Everyone is at risk for a fall, just the way everyone is at risk for infection. Everyone is. So when I’m creating my individualized plan of care, everyone is thought to be at risk for infections. Some more than others, depending on what diagnosis they may have. Whatever factors that increases their risk, you evaluate that. But I believe that a tool would be helpful to kind of putting infection control to practical use in the home care setting.”(ID221)

A number of the participants discussed how the output of a clinical risk prediction model for infection could be used to inform their care planning. It could be used to ensure that there was a specific focus on strategies for infection prevention, for those patients deemed at higher risk (Q5-Q6). One of the strategies nurses mention using to try and reduce infection risk is education of patients and their families.¹⁵ Nurses highlighted how having a quantitative value associated with a patient’s risk might alter their approach to teaching, either starting earlier in the care process or focusing their teaching interventions more precisely to the patient’s risk factors for infection (Q7-Q8). In addition, some nurses also suggested that having a risk score may encourage them to follow current infection control protocols more closely (Q9-Q10). One nurse suggested that the tool would be useful to ensure that effective infection control practices were implemented, as all patients are at risk of infection, and she would automatically assess that risk. For her, the benefit of a clinical risk prediction tool was more related to practical implications for the clinical setting than to assisting her judgment of infection risk (Q11).

The clinical risk prediction model could be seen as useful as a reminder, and for other nurses covering cases to ensure continuity and reinforcing a patient’s plan of care with the home care team (Table 2). It could act as a mechanism for communication across the healthcare team (Q12-Q13). The infection risk score could therefore be seen as “a catalyst for action. Something that has been tested and tried and will quantitatively tell the nurse that this patient is at risk for infection without all the big words and the mumbo jumbo” (ID221).

Table 2.

Informing nursing practice: Implementation, communication, and lack of utility

Mechanism for continuity and communication

Q12: “It would be a useable and functional tool in home care and also when communicating with the home medical team within the-- the home care and with the doctors as far as looking at what factors leaves this patient potentially at risk, and then continually at risk.” (ID360) Q13: “It will help us. It’ll give us like a guide. And not only just for the nurses, it will also help the HHAs as well, and any caregiver in the home who help them as well.” (ID620) Lack of benefit of a risk prediction score Q14:“I don't need a score. In my experience, I rely only on my assessments, and on the referral comments, and letters from my colleagues or doctors. I don't need a score. I just need a full information.” (ID650) Q15: “I don't need those scores to tell me that a patient is at risk because I could tell by comorbidity, and all the types of medication she's on, and the family support, the environment she's in. All these thing has to be factored together, you know what I mean?” (ID770) Q16: “I don’t know that I would particularly do anything different because I’m already just trying to-- I mean, I think every nurse’s goal is to prevent infection. No one wants their patient to get an infection.”(ID205) Q17:“Because as I mentioned earlier, the infection risk that I have in my area is really quite low compared to falls and rehospitalization.” (ID630)

However, other nurses suggested that they did not need a score to help inform their judgment, highlighting how they are able to identify a patient at risk of an infection as long as they have the right information available to them (Table 2) (Q14-Q15). In addition, while a number of nurses indicated how having a risk prediction score would alter their approach to identifying patients at risk of infection and care planning, there were also a number of nurses who highlighted that having a score would not change what they already do. These nurses suggested that as they should be using universal precautions to prevent infection with all the patients they see, having a score to identify infection risk was of little benefit (Q16). One nurse suggested that the score would not be useful because they perceived the infection risk in their area of practice to be low (Q17).

Theme 2: Operationalizing the score

Those nurses who felt that an infection risk score might be helpful to their practice had a number of ideas about how it could work, based on their experiences with other types of risk assessments and risk scores already in use across the agency. This includes common nursing risk assessments such as pressure ulcer risk assessment and fall risk assessment scores, which are required to be completed by the nurse, as well as a hospitalization risk score (which has been developed and implemented throughout the agency based on predictive models derived by agency researchers) provided to nurses at the point of care through the electronic health record.²¹

For example, one nurse suggested that it could consist of a series of questions that taken together could identify a patient’s category of risk (Q18) (Table 3). In this example, the nurse is suggesting that the system would target the collection of information to inform a risk judgment. Another nurse suggested a similar approach (using a list of options), but for them it was to help the nurse identify the particular type of infection that the patient may be at risk from, as having made the judgment themselves, a patient is at high risk (Q19).

Table 3.

Operationalizing the score: Characteristics of the information risk score

Types of Information Q18: “A series of questions to see if this patient is at risk for infection, and it would entail everything that you and I spoke of: if a patient has a wound, if the patient has a catheter, if the patient is undergoing chemotherapy, if the patient has an upper respiratory infection, if the patient lives in not such desirable environment or a very-- things like that. And each of those questions would have a number, and if the patient scores a certain number, that would put them at high risk, low risk, no risk.” (ID200) Q19: “Maybe there would be an option like, ‘Is this patient high risk of infection?’ And it will list down-- because I know in our [agency electronic health record], they only have the option, ‘Does this patient have the risk for hospitalization?’ But maybe we can also have like, ‘Is the patient risk for infection?’ And then it will list down-- there will be boxes like patient’s diabetic, patient has cancer, patient is HIV, patient noncompliant. And if we check that, then the more checks we get, then the more it predicts the higher risk for infection.” (ID480) Why a patient is at risk

Q20: “But it's not like you could highlight and see what gave those numbers. It'll just say a risk of seven, but you don't know what it is about them that made them seven.” (ID206) Q21: “So if you’re going to-- I mean, this person has a 50% risk of infection. Infection where? What type? Because in home care, the thing is when I go in, is this a wound infection? Is it a urinary tract infection? Is it a upper respiratory infection? Is it a skin infection? It’s [so large?]. So if they tell me it’s 50%, I want to know what kind of infection are we looking at.” (ID320) Integration into workflow

Q22: “I would feel it would be most helpful is that if it was in the admission questionnaire, and it builds at the end a score, and then once it reaches a certain score, then it would be for you to be alerted to these types of risk-- to a risk factor, and these protocols to go into place.”(ID215) Q23: “Continuously. I mean, both is important, but continuously is very important because you tend to get acquainted to the wound, and you see that it's good, but you forget the patient is a high risk for infection” (ID213)

Identifying the reasons why a patient may be at a higher risk of infection was deemed to be important. Nurses suggested that having a risk score without the context and the type of infection risk would not be supportive to practice (Q20-Q21). The results of the risk score need to be incorporated into existing systems and a nurses’ workflow. Some of the nurses suggested that additional questions to inform the score should be integrated into the OASIS, and others said that it should be information they receive on a patient’s referral (so prior to OASIS completion) (Q22). Nurses also felt that the risk prediction tool should be always available and updated continuously, to incorporate new information about a patient’s condition and allow for changes in risk factors over the course of the home care episode (Q23).

Nurses also highlighted how the risk prediction score should be easy to see and easy to find in the electronic health record (Table 4). Some of the nurses specifically mentioned a red-yellow-green color scheme to alert them to issues they needed to be cognizant of (Q24-Q25). One nurse suggested some sort of visualization of the information about infection risk, so that nurses could monitor it over time (Q26). Another suggested that the system could provide an alert to stop someone “going through the motions” and encouraging them to think about their practice and take remedial action if necessary (Q27). One nurse highlighted the challenges associated with implementing risk scores in practice—in that it is a new concept and, in essence, the scores may not be used effectively without targeted educational initiatives for nurses about how to incorporate information from the risk prediction tool into their care plans and practices (Q28).

Table 4.

Operationalizing the score: Specific characteristics of the information display

Q24: “But if it's like at least an exclamation point in red, I mean, that would be enough to just be there, like an oxygen alert, or fall alert, or diabetic alert on the patient's door. Just like a little sticker. Every nursing home uses different color, I guess, but it's very good.” (ID213) Q25: “I think in the profile, like if you can do red, green, or yellow. Red would be like, ‘This guy is really high risk for infection,’ like that.” (ID214)

Q26: “There could be like a scale that did it improve or got worse, so then you can monitor if there is a little scale because once you press it-- like you know when you have the OASIS and the discharge OASIS, on the right side it will say the patient takes the medication, and you put they can take it themselves, so you can see that little scale go up.” (ID219) Q27: “And I think having some type of reminder like you just mentioned, some type of alert-- I don't know how that alert would be, but at least to kind of give us that push. … Sometimes you’re just going with the motions, so at least if something's there, you'll probably take some better steps in preventing having any issues, and preventing anything from arising.” (ID204) Q28: “I don't really look at that score. I mean, I think if the patient has more than four risk factors-- if we check off more than four things on that list of risk factors, then the system will automatically say, ‘Patient is at risk for hospitalization.’ But it doesn't really kind of quantify. It doesn't really tell us a percentage. I don't think clinicians are kind of trained to use that. In nursing, we don't really use-- this predictive model is kind of new. It's a new concept. I mean, the predictive model is going to be a new concept to nurses. We don't really predict things.” (ID700)

DISCUSSION

This article describes how home care nurses perceive the utility of a clinical risk prediction model for identifying patients at risk of infection. It also highlights features that nurses felt would be important when considering how to integrate the clinical risk prediction model into existing clinical systems and workflows. The findings indicate that nurses who participated in the study would find a clinical risk prediction tool for infection useful and suggested strategies for integrating the tools into routine home care practice.

Perceived utility of a clinical risk prediction model

Clinical prediction rules are thought to reduce bias in the judgment process (in this instance, identification of a patient at high risk of infection), providing a more analytical approach to the reasoning process.⁸ The ways in which prediction rules reduce bias are thought to be through the consistent use of information to determine the probability of an event occurring, rather than through decision makers relying on judgment heuristics that can introduce error into the process.²³ Some of the nurses in our study highlighted if and how a prediction rule could be used to do this, by either helping them to prioritize information that would indicate a patient’s higher risk, or, once they had made the judgment that a patient is at higher risk of infection, indicating what the potential type of infection could be. It could also be used to alert nurses who do not normally care for patients with a high infection risk that they have a patient on their case load with an increased risk of infection. Previous studies of nurses’ use of clinical decision support system have highlighted how experience and expertise, both within a clinical context or with a decision support system itself,^24,25 can influence how the system is used. Specifically, nurses have been found to use clinical decision support systems as a “second opinion” or reassurance for a judgment that they have already taken, a finding that was also reflected in our study results.²⁵ Overall, nurses reported that using information from the risk prediction tool could be helped to support their judgment process and thereby encourage more S2 thinking, rather than relying on their experience or unconscious processes (S1 thinking).

Nurses in the study also highlighted how the results of a clinical prediction rule could be used to help inform their care planning for individual patients. A number of nurses discussed this in terms of patient and caregiver education, an intervention that has been identified through previous research as a way for nurses to mitigate infection risk.¹⁵ This could be in terms of helping them identify needs for patient and caregiver education to target specific risks for infection, or commencing education earlier in their series of home care visits with a patient. However, they also highlighted how the predictive risk model output needs to be more than just a “score” out of context; without an in-depth knowledge of the reasons why a patient was at higher risk of infection, it would not help their practice. A previous study of nurses’ use of pain assessment scores highlighted a similar issue; the use of scores is relative to the context of the patient, the care setting, and the reason for documenting the score in the first place.²⁶ In order to ensure a reduction in cognitive load,⁷ and that the clinical prediction rule output is a “catalyst for action,” we suggest considering linking risk scores with potential care interventions, and educational initiatives for nurses on how to use that information in practice, in the final intervention design. In addition, some nurses highlighted the importance of a clinical risk prediction tool to enable effective communication, which can help provide continuity of care across patients. This is an important issue, as care continuity has been associated with lower hospitalization rates and improved outcomes in home care patients.²⁷

While the majority of nurses were positive toward the potential utility of the clinical risk prediction tool in practice, there were a minority of nurses who did not feel that it would be a useful addition to their practice. These nurses highlighted how they were confident in their own judgment skills in terms of identifying patients at increased infection risk. Despite this, we suggest that nurses may find having a formal clinical risk prediction score useful for communicating with other healthcare professionals, or to justify additional resources, given the utility of such tools such as checklists as a way of communicating risk in other areas of clinical practice.²⁸ In addition, having a risk score may also help highlight the importance of infection control procedures, to home care nurses, patients, and their carers, which could lead to improved adherence to infection control procedures. Although there is some evidence to suggest that, in general, clinicians’ judgments are often as good as a risk prediction model, there is also significant evidence to suggest that often such models outperform clinical judgment.¹⁰ Further research is required to compare the output of the clinical risk prediction tool developed in our previous research⁶ with home care nurses’ judgment, to ensure that the tool does have additional predictive validity.

Nurses in our study also highlighted the tensions in practice, in which infection prevention and control should be applied to all patients in home care. While some nurses indicated that having a risk prediction score may change their adherence to infection control interventions (eg, hand hygiene or face coverings), meaning that they felt that the score would help inform their practice, others pointed out that they would use the same infection control procedures with all patients regardless of their risk. Previous studies have highlighted that while home care nurses have positive attitudes toward infection prevention and control, and good knowledge and self-reported adherence to recommended infection control practices,²⁹ their actual adherence to interventions such as hand hygiene is suboptimal.³⁰ Although it is unclear if or how a clinical risk prediction tool would impact on universal infection control interventions performed by home care nurses, there may be some benefit in focusing nurses’ attention on infection risk as a way of improving overall adherence to infection control behaviors.

Integrating a clinical risk prediction tool into practice

In order to ensure that a clinical risk prediction tool is useful in clinical practice, it needs to be easy to use, integrated into clinician workflow, available at the point of decision making, and have a clear advantage over regular ways of working.^16–19 Nurses in our study indicated a variety of different design features and strategies that they felt would be important to ensure that the prediction risk tool could be used effectively in their clinical practice. This included providing information about why a patient was identified as being at a higher risk of infection, in addition to the risk score itself, and providing that score at the point of care in the electronic health record, either prior to or immediately after a patient’s admission assessment interview. The investigator developing the predictive risk model does so independently of the assessment of patient-specific factors assessed by the bedside nurse, and the risks associated with potential or competing interventions. Predictive risk models should not dichotomize predictions in clinical practice, but rather provide a propensity to support all factors considered in the decision process. The risk prediction model that has been derived from our previous work, a multivariable logistic regression,⁶ allows for patient-level estimation of the probability of an infection and the potential to disseminate patient-level factors contributing to the final calculation.³¹ Machine learning algorithms applications to health care have continued to grow in popularity. Many of these approaches have been considered black box methods due to their lack of transparency.³² Given nurses’ need for specific information related to the components of an information risk model, we suggest that recent machine learning methods such as LIME (local interpretable model-agnostic explanations) and the application of game theory Shapley values could be used to further explain why a patient may be at high risk of infection.^32–34 Nurses also highlighted the relevance of having visualizations, either in terms of color coding associated with levels of risk or the ability to see trends in risk information over time. Both of these features have been identified in previous research co-designing dashboards with home care nurses³⁵ as a way for them to easily and efficiently access key information to inform care. Finally, one nurse indicated that an alert or a notification to indicate when a patient was at high risk of infection would also be helpful.

Study limitations

The study was conducted in 1 large home care agency in the United States, which is located in an urban area and has a diverse patient population and workforce. We have provided details on the context and setting of the study; however, it may be that our findings are not transferable to other home care agencies. The nurses in our study have had prior exposure to clinical risk prediction models to inform their practice (through existing tools in use in the agency); responses from nurses who have not had this exposure may be different.

CONCLUSION

It is important to evaluate the potential feasibility and acceptability of a clinical prediction model in clinical practice settings to inform the intervention design and implementation strategy.⁹ In this study, we outlined how a clinical risk prediction model for identifying patients at risk of infection is perceived by home care nurses, and sought to inform how the output of the model could be integrated into clinical workflow. Our findings suggest that nurses may find a tool useful for their practice, and that it would be more meaningful with features that include the context around the reasons why a patient was deemed to be at high risk. The results of this study will be used to inform the development of the clinical risk prediction tool as an intervention for implementation in home care settings.

FUNDING

This project was supported by Agency for Healthcare Research and Quality grant number R01HS024723 (to DD, JS, DR, MVD, and CB). The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.

AUTHOR CONTRIBUTIONS

DD, DR, MVD, CB, and JS contributed to conceptualization. DD, DR, MT, MVD, and JS contributed to methodology. DD, DR, MT, and MVD contributed to formal analysis. MT contributed to investigation. DD wrote the original draft of the article and DR, MT, MVD, JS, CB, and JS All authors were involved in writing, reviewing, and editing the article. DD supervised the study. MVD and JS were involved in project administration. DD, DR, MVD, CB, and JS were involved in funding acquisition.