Abstract
Objectives
The Computer-Aided Risk Score for Mortality (CARM) estimates the risk of in-hospital mortality following acute admission to the hospital by automatically amalgamating physiological measures, blood tests, gender, age and COVID-19 status. Our aims were to implement the score with a small group of practitioners and understand their first-hand experience of interacting with the score in situ.
Design
Pilot implementation evaluation study involving qualitative interviews.
Setting
This study was conducted in one of the two National Health Service hospital trusts in the North of England in which the score was developed.
Participants
Medical, older person and ICU/anaesthetic consultants and specialist grade registrars (n=116) and critical outreach nurses (n=7) were given access to CARM. Nine interviews were conducted in total, with eight doctors and one critical care outreach nurse.
Interventions
Participants were given access to the CARM score, visible after login to the patients’ electronic record, along with information about the development and intended use of the score.
Results
Four themes and 14 subthemes emerged from reflexive thematic analysis: (1) current use (including support or challenge clinical judgement and decision making, communicating risk of mortality and professional curiosity); (2) barriers and facilitators to use (including litigation, resource needs, perception of the evidence base, strengths and limitations), (3) implementation support needs (including roll-out and integration, access, training and education); and (4) recommendations for development (including presentation and functionality and potential additional data). Barriers and facilitators to use, and recommendations for development featured highly across most interviews.
Conclusion
Our in situ evaluation of the pilot implementation of CARM demonstrated its scope in supporting clinical decision making and communicating risk of mortality between clinical colleagues and with service users. It suggested to us barriers to implementation of the score. Our findings may support those seeking to develop, implement or improve the adoption of risk scores.
Keywords: internal medicine, intensive & critical care, qualitative research
Strengths and limitations of this study.
Our development of the Computer-Aided Risk Score for Mortality (CARM) has benefited from the input of practitioners, service users and carers with regard to its content, presentation, use and implementation.
Due to the scarcity of practitioner time during the period of COVID-19, our participant interviews were limited to n=9.
The CARM has been accessible to physicians on all medical wards. To date, only nurses working in critical outreach have had access to the score.
Introduction
Unplanned or emergency medical admissions to hospitals are common and involve patients with a broad spectrum of disease and illness severity.1 2 The appropriate early assessment and management of such admissions can be a critical factor in ensuring safe and high-quality care. A number of risk scoring systems have been developed which may support the clinical decision-making process, but few have been externally validated, implemented into routine practice and evaluated in situ.3 We have developed a computer-aided risk of in-hospital mortality score following emergency medical admission that automatically combines routinely collected, electronically recorded clinical data (described elsewhere2) to support recognition of deterioration and therefore to prevent deaths attributable to poor clinical monitoring. The area under the curve (c-statistic) for Computer-Aided Risk Score for Mortality (CARM) was 0.86, indicating good discrimination; it has been demonstrated to be more accurate than other similar risk scores3 and compares well with senior medical clinical judgement in identifying medical patients who were discharged alive or died in the hospital (survival to discharge).4 CARM amalgamates the National Early Warning Score (NEWS) 25 with routinely taken blood test results, age, gender and, more recently, COVID-19 status (to make it relevant during the pandemic).
Uniquely, concurrent to the statistical development of CARM, healthcare practitioners, service users and carers contributed to the development of the score to establish views on the potential value, unintended consequences, concerns and implementation needs of the score.6 Health technologies, such as risk scores, may falter due to poor implementation practices, poorly designed usability features and other contextual factors.6 Therefore, (1) as a preliminary to broader implementation of CARM in other hospitals sites, (2) to understand how potential users might interact with CARM in a clinical service environment and (3) to understand the contextual factors contributing to successful implementation and uptake, CARM was piloted at) York and Scarborough Teaching Hospitals NHS Foundation Trust by integrating it into the electronic patient health record and making it available (on login) to authorised users.
Objectives
Conduct a rapid in situ qualitative evaluation to understand the first-hand experience of practitioners interacting with CARM and making clinical decisions relating to individual patients in clinical practice.
Methods
Study design
Presentation of the score
CARM is expressed as a two-decimal point number from 0 (representing 0%) being the lowest to 1 (representing 100%) being the highest risk of mortality. The score sat on the ‘home’ screen of each patient record and by clicking on the score, the user would be taken to a screen showing each component of CARM (ie, age, gender, NEWS elements and individual blood results). This is illustrated in figure 1 for an anonymised patient. The time the score was generated, NEWS2 score and components are shown on the left, and the component blood tests are shown on the right. If users click the ‘chart’ button, they are taken to a trend chart of all CARM scores for the patient’s episode of admission.
Figure 1.
Presentation of CARM to practitioners. CARM, Computer-Aided Risk Score for Mortality; NEWS, National Early Warning Score; AKI, Acute Kidney Injury; BP, Blood Pressure.
Implementation of CARM at the pilot site
The practitioner group, described in table 1, was informed of its access to CARM by way of an email circulated by the chief clinical information officer/deputy medical director and clinical lead for the development of CARM (DR) on 20 November 2020. They were informed CARM would be live on 1 December 2020 and given links to publicly available, accessible video and online written information about CARM (CARSs Research – Helping clinical staff make informed decisions about risk).7 This information describes the development, purpose and use of CARM. Email instructions included (1) CARM should not be used in isolation from other sources of information; (2) CARM was intended to augment (not replace) clinical decision making; and (3) CARM relies on processing existing information to improve situational awareness, rather than introducing new sources of information.
Table 1.
Characteristics of those with access to CARM
| Practitioner group | Exposed to CARM (n) |
| Medical consultants (C) | 50 |
| Elderly medicine consultants | 15 |
| Medical specialist registrars | 30 |
| Elderly medicine specialist registrars | 7 |
| Consultant intensivists on CCOT | 14 |
| CCOT nurses | 7 |
| Total | 123 |
CARM, Computer-Aided Risk Score for Mortality; CCOT, critical care outreach team.
Interview questions
Our interview schedule was informed by the study aim, to understand the first-hand experience of practitioners exposed to CARM in their daily practice and clinical decision making. Questions included general experience and perceptions of impact, value, problems and presentation.
Setting
This study took place at YSTHNHSFT, a two-site 1200-bed academic acute care trust in the UK and one of the trusts in which CARM was developed. The trust has an in-house electronic patient record development team and a mature infrastructure tried and tested to combine NEWS2 with blood test results.
Participants
CARM was activated for physicians graded specialist registrar and above working in medical environments and for intensive care unit (ICU) consultants and nurses on the critical care outreach team (CCOT), all of whom may be called to assess and potentially admit deteriorating patients to ICU. This group comprised the potential participants of the study, and the numbers of each group with access to CARM are indicated in table 1. Our aim was to sample perspectives and experiences drawn from a range of different user groups, and we expected approximately 10–15 participants would be sufficient to achieve data saturation.8
Patient and public involvement
We have sought to codesign CARM with service users and carers as part of the project team as well as participants.6 9 In the study reported here, we had the input of three team members (GB, KD and JG) from the Service User and Carer Involvement in Research Group at the University of Bradford, which has been involved in the development of CARM over the last 4 years. Our service user team members were involved in interpretation of data from the patient perspective, reading and commenting on results and contributing ideas to include in the discussion and writing of this paper. We presented our data to the clinical governance group of the hospital to check if our results made sense from a clinical and organisational perspective.
Procedure
Once participants had been exposed to CARM for 4 months, a further email was sent by DR inviting them to participate in an interview and share their experiences of CARM. A participant information sheet about the aim of the study was attached with instructions to respond to JD or CMc. Interviews took place on the telephone (due to COVID-19 social distancing guidelines) and were audio recorded and transcribed verbatim. Two subsequent reminders were sent. All volunteers were interviewed by either JD or CMc, both experienced, postdoctoral qualitative health service researchers, neither of whom had any professional or personal knowledge or contact with the hospital staff included. Volunteers came from all three clinical groups (anaesthetist/intensivist, medical and nursing) leaving subsequent purposive sampling unnecessary. Prior to interviews, questions were answered and verbal consent was obtained. Interviews were held between May and July 2021.
Analysis
Interviews were professionally transcribed and analysed within NVivo V.12. Qualitative data analysis proceeded in line with Braun and Clarke’s six step process (in 2006), drawing on subsequent formulations of reflexive thematic analysis (RTA).10–12 RTA was selected as an approach well suited to flexible exploration of user’s perceptions and experiences, while balancing subjectivity, the analytical process and the data itself through iterative reflection.11 Data analysis proceeded in parallel with data collection until saturation was achieved (no further novel initial themes emerged). Once preliminary interviews had been completed, the research group first familiarised themselves with the transcribed text. Initial inductive coding was then undertaken by JD and CMc, initially independently, in order to generate candidate themes. Themes were identified at the semantic level initially, with subsequent refinement focusing on latent concepts. Following reflection on the independent coding, a convergent coding framework and commentary were produced by JD, CMc and JB integrating the prior codes, which were subsequently refined in preparation of the report by JD.
Results
Nine interviews were conducted and took between 16 and 45 min (mean duration 31 min).
Characteristics of the sample
There were nine participants, eight doctors and one CCOT nurse. A summary of participant characteristics is presented in table 2. Due to the small sample drawn from a single hospital, we have not linked these details to participant ID (presented in frequency table and in subsequent quotations) to preserve anonymity. Rather, we have ordered participants randomly in table 2. General medical participant-specified specialities included renal, emergency, gastroenterology and respiratory medicine.
Table 2.
Participant characteristics
| Role (experience in years) | Specialty | Exposure to/degree of adoption of CARM |
| Consultant (13) | General medicine | Views the score to confirm/challenge clinical judgement |
| Medical consultant (4) | General medicine | Checks the score after making clinical judgement |
| Consultant (23) | ICU/anaesthetics | Views the score out of professional curiosity |
| Medical consultant (18) | ICU/anaesthetics | Views CARM to confirm/challenge clinical judgement when called to assess deteriorating patients on medical ward |
| Senior nurse (1.5) | Critical outreach team | Views the score to confirm/challenge clinical opinion when assessing deteriorating patients on medical ward |
| Medical consultant (12) | ICU/anaesthetics | Views CARM when called to assess deteriorating patients on medical ward |
| Medical consultant (10) | General medicine | Views the score out of professional curiosity and to confirm clinical judgement |
| Specialist registrar (3) | General medicine | Views the score to confirm/challenge clinical judgement |
| Medical consultant (14) | General medicine | Views the score to confirm/challenge clinical judgement |
CARM, Computer-Aided Risk Score for Mortality; ICU, intensive care unit.
Findings
There were 4 themes and 14 subthemes; theme one current use (including support or challenge clinical judgement and decision making, communicating risk of mortality and professional curiosity), theme two barriers and facilitators to use (including litigation, resource needs, perception of the evidence base and strengths and limitations), theme three implementation support needs (including roll-out and integration, access and training and education) and theme four recommendations for development (including presentation and functionality and potential additional data). These are illustrated in figure 2 and presented in turn as follows with any differences between groups described within the narrative. The frequency with which participants spoke about any one theme is presented in table 3. Barriers and facilitators to use and recommendations for development featured highly across most interviews.
Figure 2.
Thematic map of practitioner views on the clinical utility and implementation support needs of CARM. CARM, Computer-Aided Risk Score for Mortality.
Table 3.
Frequency with which themes were discussed according to participant
| Participant | Current use | Barriers and facilitators to use | Implementation support needs | Recommendations for development |
| 1 | 6 | 7 | 2 | 7 |
| 2 | 1 | 6 | 7 | 8 |
| 3 | 4 | 6 | 2 | 3 |
| 4 | 7 | 8 | 7 | 9 |
| 5 | 6 | 6 | 3 | 2 |
| 6 | 4 | 8 | 0 | 5 |
| 7 | 1 | 2 | 2 | 0 |
| 8 | 8 | 9 | 3 | 5 |
| 9 | 4 | 5 | 3 | 3 |
Current use
This theme described how those practitioners currently exposed to CARM reported using it.
To support or challenge clinical judgement and decision making
Most participants explained how CARM confirmed (reassured) or prompted them to reconsider their clinical judgement and care decisions. There were two key areas where CARM supported judgement: ceiling of care decisions and escalation (eg, admission to ICU). Some reported CARM prompted earlier decisions than they might have made otherwise.
End of life, or people who are very sick. If you’ve got a, if you’ve got a CARM that’s sort of, that supports your clinical judgement and assessment, then that’s, there’s a degree of reassurance in there, so for me that is a benefit. P5
In that lady it prompted a Do Not Resuscitation discussion with her family members. I’d say the change would be that I’m making those decisions earlier. P7
As a means of communicating risk of mortality
CARM was often used as a means of communication with colleagues to support optimal patient care and for teaching purposes. However, for patients and relatives, the quantitative nature of chance of survival was considered particularly useful.
Because saying to someone, “oh, you’ve got a very slim chance of surviving,” initial reaction is often… “Well, I want everything … do what you can.” But then you explain well, actually, it’s less than 10% even if we do everything … “Oh, right, OK, I understand that now.” It sort of makes things a bit easier. P8
Out of professional curiosity
Some practitioners did not include CARM in their clinical decision making. They simply looked out of interest and considered how it might fit into their everyday practice. The reasons given were they saw the score as still being in development as part of a research project, the score had not yet been embedded into clinical systems, because only few practitioners had access to the score and because they did not have enough background information on the score to know how to use it.
I’m treating it as a research tool … I’m mindful that I can’t give it as much weight … I know it’s been validated, but in my mind it’s still a research tool, so therefore it’s not going to heavily, heavily play, necessarily, on my decisions. P3
Barriers and facilitators to use
There were a number of elements relating to CARM users saw as either a barrier or a facilitator.
Litigation
Litigation was a concern for a small number of participants; they believed acknowledging without taking action on CARM might leave them open to criticism. More often, criticism for failure to act was seen as an appropriate response; if a score indicates a risk, action should be taken.
If you write something in the notes about this and then you choose to acknowledge but not do anything about it, it’s, yeah, you can be hauled over the coals for it, couldn’t you, in a court. P4
Resource needs
Some participants expressed the possibility of CARM resulting in more patients needing intensive care and the potential for the need to clinically prioritise the use of those beds. Concerns about resources were also often mitigated with views that if deteriorating patients were identified sooner (using CARM), this may result in more efficient care (eg, earlier but shorter stays in the ICU). One participant thought if CARM identified deterioration early enough, it would result in more referrals to the critical outreach team, and this could prevent an admission to ICU:
If we need the resources … shout about it and say we need more ICU beds … [CARM may] lead to more referrals direct to the critical care team … the earlier we see them the more likely it is that we can prevent the deterioration to ever need critical care. P1
Perception of the evidence base for CARM
Some participants were unfamiliar with the evidence underpinning the CARM and questioned the validity of the score and were therefore reluctant to adopt the into their clinical decision-making CARM.
If I was, if I was looking at a score like this, I’d want to know … how it’s been validated, what groups it’s been validated in. P8
One participant was concerned CARM may overestimate mortality.
I think overestimating mortality risk, I think I have come across situations where I’ve thought that maybe it was doing that. P1
When practitioners believed the score underestimated the risk of mortality, it was when the score was considered in isolation. Most practitioners understood CARM was only one part of the clinical picture, one tool among many to support clinical decision making.
You’d need to see the whole, you know, the overall condition and trend of trajectory of the patient’s condition was. P3
Strengths and limitations
Strengths and limitations of CARM were expressed, often relative to NEWS. Participants considered CARM more accurate than NEWS but thought the less frequent calculation and presentation a limitation.
The CARM takes into, takes into account, not only just the observations but … the demographic, the age of a patient, and also the … biochemistry in there it’s not just as, as two-dimensional as the observation chart … [CARM] is able to give a more rounded overview of how mortality is predicted, not just a simple observational chart. P7
It’s not quite as dynamic a score as the NEWS score because … we’re not measuring the various blood parameters every day. P3
Implementation support needs
This theme related to the actions needed to be taken to fully implement and embed CARM into routine daily practice.
Roll-out and integration
One participant considered roll-out should be part of a large-scale research project:
The team goes for a big research grant … all-singing all-dancing off we go with a fanfare … that’s what I’d prefer. P1
Other suggestions included small group discussions and time to become familiar with the score, embedding it into practice, systems and processes.
A few of us getting together and talking about it, or [Clinical lead] or whoever … coming to work through some with you in the hospital would be the most useful. Just a gradual rollout P2
We have to get a feel for it … I think it does take a little bit of time to just sort of embed that into your, into your practice. P7
Participants were not sure what actions to take according to the score. Some participants suggested this might be dealt with through some sort of escalation policy or protocol.
What are you doing with this information? P5
Currently … it’s not part of the escalation policy. P4
Access
Some participants offered opinions on who might best access the system. Some thought anyone involved in taking and responding to physiological observations; others said due to the nature of the score (a predictor of mortality), it should be restricted to medical staff.
Because it’s to do with prognosis, I’d probably, without sounding too patronising, I’d probably keep it to medical staff. P8
The healthcare assistants do a lot of the observations, so it’s really important that they’re empowered to speak up about these things. P2
Although participants generally appreciated how the score was displayed, logging onto the system in different areas could be arduous.
I don’t always get access to the CARM score because, because, mainly because of the IT side of things. So, I’ve got to go and login on my login … if I’m going up to a ward, if I’m going to a computer that’s that I’ve not been to before, you might spend ten plus minutes logging, you know, for the computer to learn your permissions. P3
Training and education
Most participants suggested training and education to support the implementation of CARM. Content included the background to the score (including validation), how the score is calculated (the components), how it is different from the NEWS and what actions to take according to the score.
I'd like to know more about it really. P9
The background to it and how it’s calculated at and why it gives extra weight for a NEWS score [and] what do you do with that information? P4
Recommendations for development
Participants had a number of suggestions for the development of CARM.
Presentation
Some suggested having access to the CARM in different formats. Participants linked with ICU suggested it would be a good idea for the critical outreach team to see the CARM for all patients in the hospital in order to prioritise the patients they might visit.
So currently … we can filter high NEWS scores across the Trust … it would probably be quite useful … see who’s got a high-risk score and then be able to go and be a bit more proactive. P4
Several participants suggested CARM being accessible electronically via mobile telephones, tablets and other devices and in hard copy in the ward environment. Some suggested a visual ‘flag’ if CARM suggested the need for a review or a traffic light system to prioritise response.
… could be on mobile ‘phones … on the ward whiteboards P1
The traffic light system … it might actually make, draw your attention a little bit more, rather than a number on the bottom of the screen. P2
Participants particularly appreciated the option to see previous CARM scores presented as a trend graph:
Being able to look at changes over time, because to me that’s one of the most useful things. P2
Some participants suggested additional functionality to identify which of the various elements contributing to CARM (eg, which physiological element or blood test) had led to the score change. Others wanted an indication of when the score was refreshed with new data.
So, you do get patients whose physiological parameters are abnormal when their blood tests are normal. Or those blood tests are outdated … you just would have to be very clear about what the timescale was. P6
Potential additional data to include in CARM
While our focus was implementation and perceived utility, there were suggestions about additional data to include in CARM. We include this as a theme, not only because they may be relevant to the future development of CARM but also because offering practitioners the rationale for not including these elements may be important to wider-scale implementation. Diagnosis was frequently suggested, particularly if this was life limiting. Other suggestions were the addition of comorbidities, frailty data, nursing concerns and previous hospital admissions, all of which participants considered might aid mortality prediction.
Comorbidities … single organ failure, multi-organ failure or heart failure, liver failure … frailty. P8
Nursing concern … there are some Trusts that would add an extra three points [to NEWS] for nursing concern. P4
Frequency of hospitalisation. P5
Discussion
Our approach to the development of CARM has involved a process of codesign with healthcare professionals and service users.6 Previous papers discuss the development and validation2 and the potential value, unintended consequences, concerns and predicted implementation needs6 of the CARM. Building on these findings, in this study, we report the small pilot implementation and evaluation of CARM based on practitioners’ experiences of accessing it in clinical practice. The majority of the group interviewed had actively used CARM to support or challenge their decision making and to communicate about a patient’s status. Practitioners identified factors that helped or hindered their engagement with CARM along with their implementation support needs and recommendations for additional functionality of the score. Finally, practitioners suggested additional clinical data for consideration for inclusion in CARM, the potential for which we discuss further.
Our findings relating to current use of CARM, barriers and facilitators and implementation support needs feed into the next steps for implementing CARM across the current hospital environment, prior to scaling up. In developing CARM our aim was to identify and prevent deaths attributable to poor clinical monitoring. Our work here (and previously6) suggests CARM goes beyond this and supports clinical decisions relating to both escalation and ceiling of care. CARM aids communication with colleagues and service users. We identified barriers to CARM and implementation support needs we can seek to address prior to the next stage. Barriers expressed indicate poor engagement or inadequacy of our existing implementation materials. In line with previous research relating to risk scores on computerised systems,13 our participants had difficulty viewing contributing elements and had suggestions to enhance the display. Our findings illustrate the need for careful implementation of risk scores, early consideration and design of an implementation package, and the need to understand the requirements for support and training.
Some of our findings relating to implementation concur with others seeking to implement risk scores. For example, there is a modest literature exploring barriers to NEWS adoption summarised in two recent reviews and including lack of training14 15 (identified as a need by our participants) poorly designed documentation systems15 (which we sought to address through codesign). Other barriers identified by others included high workload14 15 and lack of skills.15 Smith et al13 investigated barriers to taking and recording physiological measures using an electronic system (most such studies involve paper-based systems). Barriers specific to the electronic system included difficulty viewing elements contributing to an elevated score and a poor display format.13 Similarly, our participants suggested a bolder means of presenting the score (eg, traffic lights) and clearer links to the elements contributing CARM. Such barriers exist with non-mortality-related computer risk scores and in other environments; for example, implementation needs relating to an infection risk score in care homes were similar to ours, including training and education needs and challenges in integrating the score into existing systems.16 This suggests that findings reported here may be transferrable to the implementation of other risk scores.
There were strengths and limitations to this study. To the best of our knowledge, CARM is the only published risk score to have involved service providers and users (as both research partners and participants) in its development as well as implementation. We did all possible to follow good practice guidelines for coworking including working as equals and meaningful, early engagement.17–20 We have received feedback from our service user colleagues,21 and we report here and elsewhere6 on the impact of practice and service user views on the score. Because of the clear value that involvement has added to both the development of CARM and in this study, we will consider formal evaluation of codesign in follow-up projects where involvement will continue to be the bedrock to our work. A population of only 123 potential participants (those with access to CARM), all of whom are busy clinicians, during a global pandemic made recruitment challenging, and we ceased recruitment after interviewing nine participants. The implementation of complex interventions requires preliminary evaluation of the intervention within the context in which it is implemented and how it contributes to support decision making in the real world.22 We sought to achieve this through flexible approaches to the timings of interviews and acknowledgement of the potential need to reschedule due to clinical shift changes or pressures. We met the balance of gaining meaningful data while avoiding taking more time than necessary from potential participants by concurrent data analysis and collection and vigilance in abiding to the guiding principles of data saturation.23 The only nurses exposed to CARM were critical outreach nurses; we did not include nurses working in other medical settings. Our findings suggest CARM has use for nurses, and our future work will include this group; however, the transferability of our findings is limited to selected groups. Finally, although we have captured self-reported exposure/degree of adoption with CARM (table 2) and we appear to have a range of responses, there may have been an element of selection bias whereby those volunteering for interviews were those the most engaged with CARM.
Conclusion
Our in-situ evaluation of the pilot implementation of CARM has demonstrated its scope in supporting clinical decision making and its use in communicating risk of mortality between clinical colleagues and with service users. We have gained insights into barriers to full implementation and adoption, implementation needs and suggestions for the development of the score in terms of presentation, functionality and content. Our findings may support those seeking to develop, implement or improve compliance with other risk scores (including NEWS).
Supplementary Material
Footnotes
Twitter: @judithdyson1
Contributors: MAM and DR had the original idea for this work. JD, CMc, JB and DR were the leads for this implementation/utility study. CMa led the service user contribution. JD, CMc, JB and MAM wrote the first draft of this paper. All authors (including GB, KD, JG, DR and MF) assisted in redrafting and approved the final version. MAM is the guarantor for this study.
Funding: This research is funded by the National Institute for Health Research Yorkshire and Humber Patient Safety Translational Research Centre.
Disclaimer: The views expressed in this article are those of the authors and not necessarily those of the National Health Service, the NIHR or the Department of Health and Social Care.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were involved in the design, conduct, reporting or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data availability statement
Data are available upon reasonable request. For access to data, interested parties may contact Judith.Dyson@bcu.ac.uk.
Ethics statements
Patient consent for publication
Not applicable.
Ethics approval
This study involves human participants. The protocol for the study was reviewed by Leeds West NRES Committee in October 2017 (17/YH/0367). In accordance with regional guidelines, the research ethics committee deemed the study was a service evaluation, and formal ethical review was not required. The service evaluation was registered with York Teaching Hospital NHS Foundation Trust in January 2020. The participants gave informed consent to participate in the study before taking part.
References
- 1.Downey CL, Tahir W, Randell R, et al. Strengths and limitations of early warning scores: a systematic review and narrative synthesis. Int J Nurs Stud 2017;76:106–19. 10.1016/j.ijnurstu.2017.09.003 [DOI] [PubMed] [Google Scholar]
- 2.Faisal M, Scally AJ, Jackson N, et al. Development and validation of a novel computer-aided score to predict the risk of in-hospital mortality for acutely ill medical admissions in two acute hospitals using their first electronically recorded blood test results and vital signs: a cross-sectional study. BMJ Open 2018;8:e022939. 10.1136/bmjopen-2018-022939 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Brabrand M, Folkestad L, Clausen NG, et al. Risk scoring systems for adults admitted to the emergency department: a systematic review. Scand J Trauma Resusc Emerg Med 2010;18:8. 10.1186/1757-7241-18-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Faisal M, Khatoon B, Scally A, et al. A prospective study of consecutive emergency medical admissions to compare a novel automated computer-aided mortality risk score and clinical judgement of patient mortality risk. BMJ Open 2019;9:e027741. 10.1136/bmjopen-2018-027741 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Royal College of Physicians . National Early Warning Score (NEWS): Standardising the assessment of acute illness severity in the NHS - Report of a working Party, 2012. [Google Scholar]
- 6.Dyson J, Marsh C, Jackson N, et al. Understanding and applying practitioner and patient views on the implementation of a novel automated computer-aided risk score (CARS) predicting the risk of death following emergency medical admission to hospital: qualitative study. BMJ Open 2019;9:e026591. 10.1136/bmjopen-2018-026591 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Richardson D. Helping clinical staff make informed decisions about risk. Available: https://carssresearch.org/2022
- 8.Guest G, Bunce A, Johnson L. How many interviews are enough? an experiment with data saturation and variability. Field methods 2006;18:59–82. [Google Scholar]
- 9.Faisal M, Richardson D, Scally A, et al. Performance of externally validated enhanced computer-aided versions of the National early warning score in predicting mortality following an emergency admission to hospital in England: a cross-sectional study. BMJ Open 2019;9:e031596. 10.1136/bmjopen-2019-031596 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Braun V, Clarke V. Thematic analysis. In: Cooper H, Camic PM, Long DL, et al., eds. Apa Handbook of research methods in psychology, vol 2: research designs: quantitative, qualitative, neuropsychological, and biological. Washington, DC: American Psychological Association[Google Scholar], 2012: 57–71. [Google Scholar]
- 11.Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc Health 2019;11:589–97. 10.1080/2159676X.2019.1628806 [DOI] [Google Scholar]
- 12.Byrne D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual Quant 2022;56:1391–412. 10.1007/s11135-021-01182-y [DOI] [Google Scholar]
- 13.Smith D, Cartwright M, Dyson J, et al. Barriers and enablers of recognition and response to deteriorating patients in the acute hospital setting: a theory-driven interview study using the theoretical domains framework. J Adv Nurs 2021;77:2831–44. 10.1111/jan.14830 [DOI] [PubMed] [Google Scholar]
- 14.Olsen SL, Søreide E, Hillman K, et al. Succeeding with rapid response systems - a never-ending process: A systematic review of how health-care professionals perceive facilitators and barriers within the limbs of the RRS. Resuscitation 2019;144:75–90. 10.1016/j.resuscitation.2019.08.034 [DOI] [PubMed] [Google Scholar]
- 15.Wood C, Chaboyer W, Carr P. How do nurses use early warning scoring systems to detect and act on patient deterioration to ensure patient safety? A scoping review. Int J Nurs Stud 2019;94:166–78. 10.1016/j.ijnurstu.2019.03.012 [DOI] [PubMed] [Google Scholar]
- 16.Dowding D, Russell D, McDonald MV, et al. "A catalyst for action": Factors for implementing clinical risk prediction models of infection in home care settings. J Am Med Inform Assoc 2021;28:334–41. 10.1093/jamia/ocaa267 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Greenhalgh T, Jackson C, Shaw S, et al. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q 2016;94:392–429. 10.1111/1468-0009.12197 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Involve . Guidance on co-producing a research project. involve. Available: https://www.invo.org.uk/posttypepublication/guidance-on-co-producing-a-research-project/2021
- 19.Flinders M, Wood M, Cunningham M. The politics of co-production: risks, limits and pollution. Evidence Policy 2016;12:261–79. 10.1332/174426415X14412037949967 [DOI] [Google Scholar]
- 20.Shimmin C, Wittmeier KDM, Lavoie JG, et al. Moving towards a more inclusive patient and public involvement in health research paradigm: the incorporation of a trauma-informed intersectional analysis. BMC Health Serv Res 2017;17:1–10. 10.1186/s12913-017-2463-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.University of Bradford . Helping clinical staff make informed decisins about risk.
- 22.Skivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of medical Research Council guidance. BMJ 2021;374:n2061. 10.1136/bmj.n2061 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health 2010;25:1229–45. 10.1080/08870440903194015 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data are available upon reasonable request. For access to data, interested parties may contact Judith.Dyson@bcu.ac.uk.