Abstract
Background
During the pandemic, remote consultations have become the norm for assessing patients with signs and symptoms of COVID-19 to decrease the risk of transmission. This has intensified the clinical uncertainty already experienced by primary care clinicians when assessing patients with suspected COVID-19 and has prompted the use of risk prediction scores, such as the National Early Warning Score (NEWS2), to assess severity and guide treatment. However, the risk prediction tools available have not been validated in a community setting and are not designed to capture the idiosyncrasies of COVID-19 infection.
Objective
The objective of this study is to produce a multivariate risk prediction tool, RECAP-V1 (Remote COVID-19 Assessment in Primary Care), to support primary care clinicians in the identification of those patients with COVID-19 that are at higher risk of deterioration and facilitate the early escalation of their treatment with the aim of improving patient outcomes.
Methods
The study follows a prospective cohort observational design, whereby patients presenting in primary care with signs and symptoms suggestive of COVID-19 will be followed and their data linked to hospital outcomes (hospital admission and death). Data collection will be carried out by primary care clinicians in four arms: North West London Clinical Commissioning Groups (NWL CCGs), Oxford-Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC), Covid Clinical Assessment Service (CCAS), and South East London CCGs (Doctaly platform). The study involves the use of an electronic template that incorporates a list of items (known as RECAP-V0) thought to be associated with disease outcome according to previous qualitative work. Data collected will be linked to patient outcomes in highly secure environments. We will then use multivariate logistic regression analyses for model development and validation.
Results
Recruitment of participants started in October 2020. Initially, only the NWL CCGs and RCGP RSC arms were active. As of March 24, 2021, we have recruited a combined sample of 3827 participants in these two arms. CCAS and Doctaly joined the study in February 2021, with CCAS starting the recruitment process on March 15, 2021. The first part of the analysis (RECAP-V1 model development) is planned to start in April 2021 using the first half of the NWL CCGs and RCGP RSC combined data set. Posteriorly, the model will be validated with the rest of the NWL CCGs and RCGP RSC data as well as the CCAS and Doctaly data sets. The study was approved by the Research Ethics Committee on May 27, 2020 (Integrated Research Application System number: 283024, Research Ethics Committee reference number: 20/NW/0266) and badged as National Institute of Health Research Urgent Public Health Study on October 14, 2020.
Conclusions
We believe the validated RECAP-V1 early warning score will be a valuable tool for the assessment of severity in patients with suspected COVID-19 in the community, either in face-to-face or remote consultations, and will facilitate the timely escalation of treatment with the potential to improve patient outcomes.
Trial Registration
ISRCTN registry ISRCTN13953727; https://www.isrctn.com/ISRCTN13953727
International Registered Report Identifier (IRRID)
DERR1-10.2196/29072
Keywords: COVID-19 severity, risk prediction tool, early warning score, hospital admission, primary care, electronic health records
Introduction
Overview
During 2020, it became clear that assessment of the severity of COVID-19 infection required clinical tools specific to the condition and that repurposing tools such as the National Early Warning Score (NEWS2), designed for the early diagnosis of sepsis, would not be safe clinical practice [1]. The management of COVID-19 by clinicians is challenged by uncertainty about the disease progression [2]. There is evidence that a small percentage of patients present a dramatic deterioration of clinical status around the 8th to 10th day of disease, often associated with unperceived low oxygen saturations (known as “silent hypoxia”) that may require hospital and intensive care unit (ICU) admissions [3-5]. The inability to predict which patients will experience clinical deterioration adds an additional level of complexity to the clinical challenge and diagnostic uncertainty that general practitioners (GPs) have faced during the pandemic, particularly as most of the consultations are carried out remotely (usually by telephone and occasionally by video) to minimize the risk of transmission [6].
It was initially suggested that NEWS2 could be used to assess severity of patients with COVID-19 [7]. NEWS2 is calculated from patient’s temperature, pulse rate, respiratory rate, systolic blood pressure, pulse oximetry reading, and presence of new onset of acute confusion [8]. It is commonly used in hospital settings and ambulance service prior to transfer to hospital to assess the risk of deterioration of a patient [9]. However, NEWS2 seems to be a late indicator of decompensation, typically triggering within the last 12 hours before a transfer to ICU is considered necessary and, therefore, this limits its application and validity in a primary care or community care setting where an earlier warning would be preferred [9,10].
The Roth score (originally developed as a measure of breathlessness in cardiopulmonary disease [11]) was briefly considered by the Royal College of General Practitioners (RCGP) as possibly useful in the assessment of breathlessness when assessing patients with signs and symptoms of COVID-19 [12]. However, a rapid literature review concluded that the Roth score might have a low sensitivity (ie, a normal score in patients with “silent hypoxia”), and therefore should not be used by GPs when assessing patients over the phone or in video consultations [13].
Justification and Study Objective
This new condition and the forced shift toward remote consultations during the pandemic have increased the challenges and uncertainty commonly faced in general practice [6]. Primary care clinicians need a tool to guide the management of patients with suspected COVID-19 to be able to identify those whom they can reassure, those that need monitoring, and those that require urgent further assessment or referral to hospital. Even though the validity of NEWS2 for this purpose was a subject of intense debate during the height of the first COVID-19 wave, the score is still being used by primary care clinicians to assess patients prior to transfer to hospital [9]. The use of NEWS2 outside the hospital setting has not been validated, and it was not designed to capture the idiosyncrasies of COVID-19 infection. Therefore, there is need to develop an early warning score that incorporates key features of acute COVID-19 and that can be safely used by GPs when assessing patients remotely [14].
We reviewed the literature on COVID-19 early warning scores, then conducted a series of focus groups with 72 primary care clinicians (mostly GPs and including advanced nurse practitioners and paramedics) to derive elements that might form part of a suitable score, value sets, and appropriate SNOMED terms [15]. This paper describes the process of quantitative development and validation of the Remote COVID-19 Assessment in Primary Care (RECAP) score. The objective was to produce a multivariate risk prediction tool to facilitate the early identification, by primary care physicians and other clinicians working in the community, of those patients with COVID-19 that are at higher risk of becoming severely ill and inform the early escalation of their treatment, while also reducing unnecessary referrals in low-risk patients, with the aim of improving patient outcomes.
Methods
Study Design
This primary care data linkage study follows a prospective cohort observational design, whereby patients presenting in primary or community care with signs and symptoms suggestive of COVID-19 will be followed and their data linked with hospital outcomes, particularly focusing on hospital admission, ICU admission, and death. For data collection purposes, the initial set of items identified in earlier qualitative work [15], known as RECAP-V0, will be integrated into an electronic template to be used by primary care physicians (see Figure 1 for a summary of items included in RECAP-V0). This will enable the standardized recording of patients’ signs and symptoms and subsequent linkage with hospital and mortality data. Data collected will be used to develop and validate a multivariate regression model to predict hospital admission, ICU admission, and death.
Figure 1.
Summary of RECAP-V0 items. Source: [15]. RECAP: Remote COVID-19 Assessment in Primary Care.
Data Collection
Recruitment
The development of the RECAP score will require the use of primary and secondary data. The collection of patients’ signs and symptoms as they present in primary care requires the involvement of primary care clinicians, who will be asked to assess those patients with a clinical diagnosis of suspected COVID-19 using the RECAP electronic template.
The recruitment of clinicians (study sites) and patients (study participants) will be carried out by four different arms depending on clinician and participant location and service used to seek medical care:
North West London (NWL) Clinical Commissioning Groups (CCGs) arm: this arm has its own integrated linked database (Whole Systems Integrated Care [WSIC]) and a secure environment (Imperial’s Clinical Analytics, Research and Evaluation [iCARE] secure environment) to hold the data. Recruitment of practices will be facilitated by the NWL clinical research network (CRN). General practitioners will use EMIS [16] or TPP SystmOne [17] electronic health record systems to capture patients’ data.
RCGP Research and Surveillance Centre (RSC) arm: this is a national network of practices within the RCGP developed to contribute with data for disease surveillance and research [18], which is held in the Oxford RCGP Clinical Informatics Digital Hub (ORCHID) secure environment [19]. Subject to the patient’s consent, data from RSC network practices (collected from computerized medical record systems EMIS or TPP SystmOne, the United Kingdom’s most used systems, using Ardens RECAP electronic templates [20]) will be pseudonymized and extracted via a Wellbeing Software extraction system and linked to outcomes.
Covid Clinical Assessment Service (CCAS) arm: this service is organized within the National Health Service (NHS) 111 Online service (managed by South Central Ambulance Service) for the clinical assessment and management of patients with a clinical diagnosis of suspected COVID-19. It is staffed by general practitioners and uses the Adastra electronic health record system [21]. Upon patients’ consent, the data collected will be transferred to ORCHID and linked to hospital outcomes.
Doctaly arm: this private health care platform has been commissioned by South East London CCGs to provide a home monitoring service for patients with a diagnosis of COVID-19 (positive result in laboratory test) in South East London. Patients’ medical history and assessment data are collected using a chatbot via the WhatsApp mobile app. The questions asked via the Doctaly chatbot were designed to reflect the same concepts as the RECAP-V0 set. Data collected will be also transferred to the Oxford secure environment and linked to outcome data.
Figure 2 below depicts study data sources and data flow. Primary care data collected by practices in NWL and held in iCARE are already linked to hospital outcomes (ie, hospital admission, ICU admission, and death). Data held in the University of Oxford secure environment (RCGP RSC, CCAS, and Doctaly data) will be linked to outcome data contained in the Hospital Episode Statistics (HES) and Office of National Statistics (ONS) databases using an encrypted NHS number. Hospital admission and mortality data are available in HES and ONS; however, ICU admission information is not available.
Figure 2.
Data flowchart. CCG: Clinical Commissioning Group; NHS: National Health Service; NWL: North West London; ORCHID: Oxford RCGP Clinical Informatics Digital Hub; RCGP: Royal College of General Practitioners; RSC: Research and Surveillance Centre; SE: South East.
Selection Criteria
Our main cohort includes patients clinically diagnosed with COVID-19 that are being assessed and managed in primary care. Additional cohorts include patients with signs and symptoms suggestive of COVID-19 assessed by the NHS 111 CCAS and patients with established COVID-19 that are assessed as part of a primary care–led home monitoring service (Doctaly).
In the NWL, RCGP RSC, and CCAS arms, participants will be identified by primary care clinicians and enrolled in the study if they satisfy the following inclusion criteria:
The patient is willing and able to provide informed consent for data linkage (exceptions are described in detail in the Overview section of the Results)
The patient has signs and symptoms that are judged by the clinician to be suggestive of acute COVID-19 and time since onset of symptoms is ≤14 days.
The participant is 18 years of age or older.
The clinician is able to use the electronic template that contains the RECAP codes.
Data collected by the clinician can be linked to the following hospital outcomes: hospital admission, ICU admission (only for NWL CCGs arm data), and hospital outcome (either discharge or cause of death).
For data collected in South East London CCGs (Doctaly) arm, the selection criteria consist of participant age (ie, 18 years old or older) and having a data sharing or consent procedure in place, since the other criteria are already satisfied (ie, patients are offered home monitoring after receiving a positive result from a COVID-19 test and the monitoring tool was specifically designed to include RECAP codes).
Template Development
In order to collect primary data from primary care or community care settings, the RECAP-V0 items that captured patients’ signs and symptoms along with other characteristics (sociodemographic information and comorbidities) are transferred into an electronic template using SNOMED and Read codes. These codes have been identified by the study team and collaborators and have been reviewed by NHSX, NHS England, and the UK Faculty of Clinical Informatics. The templates have been deployed for COVID-19 management via electronic health record systems—such as Ardens EMIS and SystmOne, TPP SystmOne, or Adastra—used by clinicians in GP practices, COVID-19 hubs, and CCAS, or via the patient-facing platform Doctaly. This will enable the collection of patients’ signs and symptoms in large data sets that will be stored in two secure environments (ORCHID and iCARE secure environments).
Sample Size
A total of 2880 participants will be necessary to develop a model with a minimum 85% specificity, assuming 10% prevalence of hospital admission and 6% missing data. We will split the sample into two consecutive groups, taking the first 50% of participants’ data for model development and the last 50% for model validation. CCAS will also collect 2880 participants as we wish to explore the hypothesis that, on account of case mix and spectrum bias, patients already triaged to the national service may require a separate model. We will then separately develop and validate a model for CCAS. Doctaly will provide an additional validation data set for RECAP-V1 score.
Data Analysis
Overview
A detailed statistical analysis plan (SAP) written before inspecting the data will be followed for analysis. The SAP provides a detailed description of data handling, RECAP-V1 model development and validation, and any planned secondary outcomes analysis. Given the complexity of issues to be addressed, including missing data not at random, potential correlations between clinical measurements; regression models and machine learning; and the relationships between the four different data sets, the SAP will be the subject of a separate article.
RECAP-V1 Early Warning Score Development and Validation
We will use multivariate logistic regressions to develop and validate the score. Table 1 contains a list of the items we included in the RECAP-V0 electronic template along with their SNOMED codes that will be used as inputs in the model.
Table 1.
RECAP-V0 template items.
| Variable name | Measurement level | Description/parameter | Concept ID |
| Heart rate | Continuous |
|
|
| Respiratory Rate | Continuous |
|
|
| Shortness of breath | Nominal |
|
|
| Trajectory of breathlessness | Nominal |
|
|
| Oxygen saturation (rest) | Continuous |
|
|
| Oxygen saturation (exertion) | Continuous |
|
|
| Level of tiredness | Nominal |
|
|
| Temperature | Continuous |
|
|
| Feeling feverish | Nominal |
|
|
| Date of onset of symptoms | Continuous |
|
|
| Muscle aches | Nominal |
|
|
| Cognitive decline | Nominal |
|
|
| COVID-19 shielding list as defined using the QCOVID score [23] | Nominal |
|
|
| Age | Continuous |
|
|
| Body mass index | Continuous |
|
|
| Patient sex | Nominal |
|
|
| Diabetes | Nominal |
|
|
| Hypertension | Nominal |
|
|
| Coronary heart disease | Nominal |
|
|
| Chronic kidney disease | Nominal |
|
|
| Ethnicity/related nationality data | Nominal |
|
|
| Participant’s consent | Nominal |
|
|
The template has been designed to support the assessment of patients via both face-to-face and remote consultations; however, we anticipate that there are certain observations, such as respiratory rate or oxygen saturation, whose recording in remote consultations may be challenging. Therefore, we included information on patients’ symptoms that could be used as a proxy of quantitative items if they were unavailable. The factors for the model (predictor variables) can then be summarized as follows: heart rate, respiratory rate or shortness of breath, trajectory of breathlessness, oxygen saturation or level of tiredness, temperature or feeling feverish, days from onset of symptoms, muscle aches, and cognitive decline. Moreover, we will extract other patient characteristics such as age, gender, body mass index, ethnicity, presence of comorbidities (eg, diabetes, hypertension, coronary heart disease, and chronic kidney disease), and whether the patient is or has been on a COVID-19 shielding list. During the conduct of the study, the QCOVID score [22] has been adopted as a measure of baseline risk and used to populate the COVID-19 shielding list in health record systems [23]. We expect that patient characteristics ought to be able to be represented by the shielding term and will test this hypothesis. Missing data will be handled with standard methodologies for the multiple imputation of missing data [24].
Regarding the outputs of the model, we are interested in hospital admission (defined as an overnight hospital stay within 28 days of onset of symptoms), ICU admission (only available in NWL’s WSIC/iCARE database), and death (either at the hospital or at home within 28 days of onset of symptoms).
We will also conduct exploratory analyses, using machine learning algorithms for outcome prediction (nonlinear classifiers) including random forest, gradient boosting, and neural networks, alongside machine learning approaches for imputation of missing data.
Results
Overview
Recruitment started in October 2020. Initially, only the NWL CCGs and RCGP RSC arms were actively recruiting. In order to engage clinicians with the study and facilitate participation, we have run three webinars or training workshops where the study team described the study objectives and deadlines, and provided a detailed description of the RECAP template and how it would be used at the clinical front line. These webinars took place in October 2020 and January 2021. The study team has been in close contact with participating practices through the Imperial College London arm office, which has overall responsibility for the project and is directly in charge of data collection in North West London, and the University of Oxford arm office, which has direct responsibility for data collection from RCGP RSC practices. The CCAS and Doctaly platform arms joined the study in February 2021.
The initial data set to be used for model development will consist of RCGP RSC and NWL arms data and will be complete by the end of March 2021; this includes the primary care data on recruited patients’ signs and symptoms collected by these two arms linked to outcomes 28 days later. Two stages of data extraction and analysis of this integrated data set have been identified: first, RECAP-V1 development using the first half of the data set will start in April 2021, and second, model validation using the second half of the data set will follow. As of March 24, 2021, we have recruited a combined sample of 3827 participants (173 active primary care practices enrolled) in these two arms. The CCAS arm started the recruitment process on March 15, 2021, and we expect to reach the desired sample size in this arm (2880 participants) by the end of May 2021. Data sharing agreements are being developed to access data that have already been collected from around 1400 participants by clinicians using the Doctaly platform. The CCAS and Doctaly platform data sets will be used to validate the RECAP-V1 model and will be analyzed independently. Once we have produced the model, and subject to findings, the RECAP-V1 score will be ready to be deployed and used by clinicians to guide the management of patients with suspected COVID-19 according to their predicted risk.
The study is sponsored by Imperial College London and ethical approval was granted by the North West-Greater Manchester East Research Ethics Committee and Health Research Authority on May 27, 2020 (Integrated Research Application System number: 283024, Research Ethics Committee reference number: 20/NW/0266). An amendment to include the CCAS and Doctaly arms was approved on February 1, 2021. Due to the low risk associated with participation in this study and the remote nature (telephone/video consultation) of most patient encounters, the review committee agreed that obtaining verbal consent for data linkage was acceptable.
To access and link retrospective data collected by the NHS 111 CCAS and Doctaly platforms in South East London (ie, data that have already been collected by the services prior to study participation) we requested the last ethics amendment submitted to be assessed under the Control of Patient Information (COPI) notice, data sharing provisions that allow public authorities and research bodies the use of COVID-19–relevant patient-level data without the need for patients’ explicit consent [25]. For NHS 111 CCAS prospective data—that is, data from patients seeking medical care after the RECAP template has been installed in Adastra—we will apply the same mechanism to seek consent that has been followed in the NWL and RCGP RSC arms, and patients in the clinical queue will receive an SMS text message with information on the study and how to participate.
Data and all appropriate documentation will be stored in accordance with General Data Protection Regulation (Data Protection Act 2018) for a minimum of 10 years after the completion of the study, including the follow-up period. Participants can withdraw from the study at any point by informing their GP or a member of the study team. They will be asked whether the data obtained before withdrawal can be retained for analysis or they would like their data to be destroyed instead.
The study was included in the National Institute of Health Research (NIHR) Clinical Research Network Portfolio (CPMS number: 45890) on September 25, 2020, and badged as NIHR Urgent Public Health Study on October 14, 2020. These measures facilitate the rapid mobilization of resources from NIHR and clinical research networks toward study dissemination and participant recruitment and help ensure that high-quality data can be collected on a timely basis (trial registration number: ISRCTN13953727).
Dissemination Plan and Patient and Public Involvement
The RECAP-V0 template has already been disseminated nationally through CRNs facilitating the standardization of clinical records of patients with COVID-19. Its use has been encouraged through webinars and invited talks arranged by CRNs. Once the risk prediction tool has been developed and validated, we will seek endorsement for it to be incorporated into the electronic template to support clinical decision making when assessing patients with COVID-19. We expect to reach wide national and international dissemination of the RECAP-V1 risk prediction tool through submission to academic journals and international conferences.
Patient and public participation has been incorporated at different stages of the project. Patients were involved, along with primary care clinicians, in the qualitative study carried out to identify the set of elements to be included in the RECAP-V0 [15]. Once the RECAP-V1 tool has been developed and validated, public participation will be sought to coproduce project lay summaries, which will be valuable to disseminate the study findings to a wider audience.
Discussion
The RECAP-V1 early warning score will, we anticipate, facilitate the stratification of the severity of patients with COVID-19 and their appropriate management and escalation of treatment. This study also promotes the standardization of assessment of patients with COVID-19, of collection of medical records, and record keeping thanks to the electronic templates developed, which can all have a positive impact in patients’ care, continuity, and safety [26]. Moreover, since November 2020, NHS England and NHS Improvement have led the establishment of the COVID Oximetry @home pathway, offered to patients with symptomatic COVID-19 who are aged 65 years or older or who are clinically extremely vulnerable to COVID-19 [27]. This service is being delivered by general practice, with referrals from NHS 111, CCAS, and hospital emergency departments, and involves an initial face-to-face or remote clinical assessment followed by monitoring of home oximetry readings for 14 days, to aid early recognition of deterioration. Items in the RECAP-V1 risk prediction tool in development are consistent with suggested clinical markers for triage on this pathway, and we anticipate that the tool will provide a unified quantitative risk score that will fit the monitoring needs of the service. Finally, we would like to emphasize the value of the study as an example of a digital clinical study, whose practice has been upheld by national research institutions on the basis of its cost-effectiveness and patient-centeredness due to the potential to recruit participants and collect large amounts of data with minimum inconvenience for the patient [28]. This is an example of leveraging the power of the NHS as a learning health system [29].
Acknowledgments
This study is funded by the Community Jameel and the Imperial College President’s Excellence Fund, the Economic and Social Research Council, the UK Research and Innovation, Health Data Research UK, the NIHR Imperial Biomedical Research Centre (the study was enabled by the Imperial Clinical Analytics Research and Evaluation (iCARE) environment and Whole Systems Integrated Care and used the iCARE and WSIC team and data resources, https://imperialbrc.nihr.ac.uk/facilities/icare), the NIHR Oxford Biomedical Research Centre and the NIHR Imperial Patient Safety Translational Research Centre. We also thank our collaborators at EMIS, NWL CCGs, TPP and Wellbeing medical systems. Finally, we would like to acknowledge patients and practices of the Oxford-RCGP RSC, NWL CCGs, CCAS and SE London CCGs, who allowed data sharing for research, and the NIHR Clinical Research Networks for their help in enrolling and liaising with practices.
Abbreviations
- CCG
Clinical Commissioning Group
- COPI
Control of Patient Information
- CRN
clinical research network
- GP
general practitioner
- HES
Hospital Episode Statistics
- iCARE
Imperial’s Clinical Analytics, Research and Evaluation database
- ICU
intensive care unit
- NEWS2
National Early Warning Score
- NHS
National Health Service
- NIHR
National Institute of Health Research
- ONS
Office of National Statistics
- ORCHID
Oxford RCGP Clinical Informatics Digital Hub
- RCGP
Royal College of General Practitioners
- RECAP
Remote COVID-19 Assessment in Primary Care
- RSC
Research and Surveillance Centre
- SAP
statistical analysis plan
- WSIC
Whole Systems Integrated Care
Appendix
Peer review document.
Peer review comments.
Footnotes
Conflicts of Interest: SdL is the Director of the RCGP RSC. All other authors declare no conflicts.
References
- 1.Carr E, Bendayan R, Bean D, Stammers M, Wang W, Zhang H, Searle T, Kraljevic Z, Shek A, Phan HTT, Muruet W, Gupta RK, Shinton AJ, Wyatt M, Shi T, Zhang X, Pickles A, Stahl D, Zakeri R, Noursadeghi M, O'Gallagher K, Rogers M, Folarin A, Karwath A, Wickstrøm KE, Köhn-Luque A, Slater L, Cardoso VR, Bourdeaux C, Holten AR, Ball S, McWilliams C, Roguski L, Borca F, Batchelor J, Amundsen EK, Wu X, Gkoutos GV, Sun J, Pinto A, Guthrie B, Breen C, Douiri A, Wu H, Curcin V, Teo JT, Shah AM, Dobson RJB. Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study. BMC Med. 2021 Jan 21;19(1):23. doi: 10.1186/s12916-020-01893-3. https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-020-01893-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Koffman J, Gross J, Etkind SN, Selman L. Uncertainty and COVID-19: how are we to respond? J R Soc Med. 2020 Jun;113(6):211–216. doi: 10.1177/0141076820930665. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Chen S, Feng H, Xu H, Huang S, Sun J, Zhou L, He J, Song W, Wang R, Li X, Fang M. Patterns of Deterioration in Moderate Patients With COVID-19 From Jan 2020 to Mar 2020: A Multi-Center, Retrospective Cohort Study in China. Front Med (Lausanne) 2020;7:567296. doi: 10.3389/fmed.2020.567296. doi: 10.3389/fmed.2020.567296. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Bickler PE, Feiner JR, Lipnick MS, McKleroy W. 'Silent' Presentation of Hypoxemia and Cardiorespiratory Compensation in COVID-19. Anesthesiology. 2021 Feb 01;134(2):262–269. doi: 10.1097/ALN.0000000000003578. https://pubs.asahq.org/anesthesiology/article-lookup/doi/10.1097/ALN.0000000000003578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wilkerson RG, Adler JD, Shah NG, Brown R. Silent hypoxia: A harbinger of clinical deterioration in patients with COVID-19. Am J Emerg Med. 2020 Oct;38(10):2243.e5–2243.e6. doi: 10.1016/j.ajem.2020.05.044. http://europepmc.org/abstract/MED/32471783. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Greenhalgh T, Koh GCH, Car J. Covid-19: a remote assessment in primary care. BMJ. 2020 Mar 25;368:m1182. doi: 10.1136/bmj.m1182. [DOI] [PubMed] [Google Scholar]
- 7.COVID-19 rapid guideline: critical care in adults. National Institute for Health and Care Excellence (NICE) 2020. Mar 20, [2021-05-11]. https://www.nice.org.uk/guidance/ng159. [PubMed]
- 8.National Early Warning Score (NEWS) 2. Royal College of Physicians (UK) [2021-03-07]. https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news-2.
- 9.Patel R, Nugawela MD, Edwards HB, Richards A, Le Roux H, Pullyblank A, Whiting P. Can early warning scores identify deteriorating patients in pre-hospital settings? A systematic review. Resuscitation. 2018 Nov;132:101–111. doi: 10.1016/j.resuscitation.2018.08.028. [DOI] [PubMed] [Google Scholar]
- 10.Finnikin S, Wilke V. What's behind the NEWS? National Early Warning Scores in primary care. Br J Gen Pract. 2020 Jun;70(695):272–273. doi: 10.3399/bjgp20X709361. http://europepmc.org/abstract/MED/32269041. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Chorin E, Padegimas A, Havakuk O, Birati EY, Shacham Y, Milman A, Topaz G, Flint N, Keren G, Rogowski O. Assessment of Respiratory Distress by the Roth Score. Clin Cardiol. 2016 Nov;39(11):636–639. doi: 10.1002/clc.22586. doi: 10.1002/clc.22586. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.RCGP clarification on the use of Roth scores in the assessment of patients with potential COVID-19. Royal College of General Practitioners. 2020. Apr 2, [2021-03-07]. https://www.rcgp.org.uk/about-us/news/2020/april/rcgp-clarification-on-the-use-of-roth-scores-in-the-assessment-of-patients-with-potential-covid-19.
- 13.Greenhalgh T. Should the Roth score be used in the remote assessment of patients with possible COVID-19? Centre for Evidence-Based Medicine, Oxford. The Centre for Evidence-Based Medicine, Oxford. [2021-05-11]. https://www.cebm.net/covid-19/roth-score-not-recommended-to-assess-breathlessness-over-the-phone/
- 14.Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, De Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma Jie, Martin GP, McLernon DJ, Andaur CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020 Apr 07;369:m1328. doi: 10.1136/bmj.m1328. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=32265220. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Greenhalgh T, Thompson P, Weiringa S, Neves AL, Husain L, Dunlop M, Rushforth A, Nunan D, de Lusignan S, Delaney B. What items should be included in an early warning score for remote assessment of suspected COVID-19? qualitative and Delphi study. BMJ Open. 2020 Nov 12;10(11):e042626. doi: 10.1136/bmjopen-2020-042626. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.EMIS Health. [2021-03-31]. https://www.emishealth.com/
- 17.Clinical software to revolutionise healthcare. The Phoenix Partnership (TTP) [2021-03-31]. https://www.tpp-uk.com/
- 18.RCGP Research and Surveillance Centre (RSC) Royal College of General Practitioners. [2021-03-07]. https://www.rcgp.org.uk/clinical-and-research/our-programmes/research-and-surveillance-centre.aspx.
- 19.de Lusignan S, Lopez Bernal J, Byford R, Amirthalingam G, Ferreira F, Akinyemi O, Andrews N, Campbell H, Dabrera G, Deeks A, Elliot AJ, Krajenbrink E, Liyanage H, McGagh D, Okusi C, Parimalanathan V, Ramsay M, Smith G, Tripathy M, Williams J, Victor W, Zambon M, Howsam G, Nicholson BD, Tzortziou Brown V, Butler CC, Joy M, Hobbs FDR. Influenza and Respiratory Virus Surveillance, Vaccine Uptake, and Effectiveness at a Time of Cocirculating COVID-19: Protocol for the English Primary Care Sentinel System for 2020-2021. JMIR Public Health Surveill. 2021 Feb 19;7(2):e24341. doi: 10.2196/24341. https://publichealth.jmir.org/2021/2/e24341/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ardens Health Informatics Ltd. [2021-03-31]. https://www.ardens.org.uk/
- 21.Adastra Clinical Patient Management. [2021-03-31]. https://www.oneadvanced.com/solutions/adastra/
- 22.Clift AK, Coupland CAC, Keogh RH, Diaz-Ordaz K, Williamson E, Harrison EM, Hayward A, Hemingway H, Horby P, Mehta N, Benger J, Khunti K, Spiegelhalter D, Sheikh A, Valabhji J, Lyons RA, Robson J, Semple MG, Kee F, Johnson P, Jebb S, Williams T, Hippisley-Cox J. Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ. 2020 Oct 20;371:m3731. doi: 10.1136/bmj.m3731. http://www.bmj.com/lookup/pmidlookup?view=long&pmid=33082154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.COVID-19 Population Risk Assessment. NHS Digital. [2021-03-21]. https://digital.nhs.uk/coronavirus/risk-assessment/population.
- 24.Lang KM, Little TD. Principled Missing Data Treatments. Prev Sci. 2018 Apr;19(3):284–294. doi: 10.1007/s11121-016-0644-5. [DOI] [PubMed] [Google Scholar]
- 25.Control of patient information (COPI) notice. NHS Digital. [2021-03-21]. https://digital.nhs.uk/coronavirus/coronavirus-covid-19-response-information-governance-hub/control-of-patient-information-copi-notice.
- 26.Tanner C, Gans D, White J, Nath R, Pohl J. Electronic health records and patient safety: co-occurrence of early EHR implementation with patient safety practices in primary care settings. Appl Clin Inform. 2015;6(1):136–47. doi: 10.4338/ACI-2014-11-RA-0099. http://europepmc.org/abstract/MED/25848419. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.COVID Oximetry @home - digital and data services. NHS Digital. [2021-03-09]. https://digital.nhs.uk/coronavirus/covid-oximetry-at-home-digital-and-data-services.
- 28.Inan OT, Tenaerts P, Prindiville SA, Reynolds HR, Dizon DS, Cooper-Arnold K, Turakhia M, Pletcher MJ, Preston KL, Krumholz HM, Marlin BM, Mandl KD, Klasnja P, Spring B, Iturriaga E, Campo R, Desvigne-Nickens P, Rosenberg Y, Steinhubl SR, Califf RM. Digitizing clinical trials. NPJ Digit Med. 2020;3:101. doi: 10.1038/s41746-020-0302-y. doi: 10.1038/s41746-020-0302-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Scobie S, Castle-Clarke S. Implementing learning health systems in the UK NHS: Policy actions to improve collaboration and transparency and support innovation and better use of analytics. Learn Health Syst. 2020;4(1):e10209. doi: 10.1002/lrh2.10209. doi: 10.1002/lrh2.10209. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Peer review document.
Peer review comments.