Abstract
Introduction:
Late recognition of in-hospital deterioration is a source of preventable harm. Emergency transfers (ET), when hospitalized patients require intensive care unit (ICU) interventions within 1 hour of ICU transfer, are a proximal measure of late recognition associated with increased mortality and length of stay (LOS). Applying diagnostic process improvement frameworks, we aimed to identify missed opportunities for improvement in diagnosis (MOID) in ETs and evaluate their association with outcomes.
Methods:
A single-center retrospective cohort study of ETs, January 2015-June 2019. ET criteria included intubation, vasopressor initiation, or ≥60 ml/kg fluid resuscitation 1 hour before to 1 hour after ICU transfer. The primary exposure was presence of MOID, determined using SaferDx. Cases were screened by an ICU and non-ICU physician. Final determinations were made by an interdisciplinary group. Diagnostic process improvement opportunities were identified. Primary outcomes were in-hospital mortality and post-transfer LOS, analyzed by multivariable regression adjusting for age, service, deterioration category, and pre-transfer LOS.
Results:
MOID was identified in 37 of 129 ETs (29%, 95%CI 21-37%). Cases with MOID differed in originating service, but not demographically. Recognizing the urgency of an identified condition was the most common diagnostic process opportunity. ET cases with MOID had higher odds of mortality (odds ratio 5.5, 95%CI 1.5 - 20.6, p 0.01) and longer post-transfer LOS (rate ratio 1.7, 95%CI 1.1 – 2.6, p 0.02).
Conclusion:
MOID are common in ETs and associated with increased mortality risk and post-transfer LOS. Diagnostic improvement strategies should be leveraged to support earlier recognition of clinical deterioration.
Keywords: Diagnostic Decision Making, Patent Safety, Pediatric Hospital Medicine, Rapid Response Team, Clinical Deterioration
Introduction
Among hospitalized patients, late recognition of clinical deterioration is a common and significant source of preventable harm.1 In the pediatric health services literature, emergency transfers (ET) have been identified as a proximal metric of late recognition of clinical deterioration.2,3 ETs, in-hospital deterioration events requiring intensive care unit (ICU) interventions within 1 hour of ICU transfer, are associated with increased mortality and length of stay (LOS).2,3 To date, efforts to promote early recognition of clinical deterioration and reduce ETs have focused on improving automated detection systems, situation awareness, and rapid response system infrastructure with variable success.1,4-9 Incorporating alternative methodologic frameworks to study and mitigate deterioration events could provide new directions for improvement.
Diagnostic process improvement frameworks are increasingly recognized as important domains of quality and safety, and these frameworks could provide an alternative approach to preventing unrecognized clinical deterioration.10-12 Diagnostic errors, commonly referred to as “missed opportunities to improve diagnosis” (MOID), are defined as missed opportunities to make the correct or timely diagnosis. MOID is thought to be frequent and associated with negative outcomes in pediatric and adult hospital settings.10,13 13-15 However, identifying MOID can be challenging and resource-intensive, which limits the potential for using MOID to inform systems learning and improvement.
ETs, which are trackable in the electronic health record (EHR) and involve late recognition of a potentially unexpected clinical change, could provide an enriched cohort to identify and study MOID in clinical deterioration.3,16 While diagnostic improvement frameworks have demonstrated reliability and validity in acute pediatric settings, the incidence of MOID and potential improvement opportunities associated with ETs have not been characterized.16
We utilized diagnostic process improvement tools to evaluate for MOID in an existing cohort of ETs, with the aims of 1) identifying and describing the incidence of MOID amongst ETs; 2) identifying diagnostic process opportunities to inform future improvements in ET detection and prevention; and 3) evaluating the association of MOID with outcomes in the ET cohort. This represents an important first step in evaluating whether diagnostic process improvement frameworks could be used to prevent unrecognized clinical deterioration and, ultimately, avert patient harm.
Methods
Study Design
In this single center retrospective cohort study, we utilized diagnostic process improvement tools to evaluate for MOID in unplanned transfers from non-cardiac medical or surgical wards (e.g., not scheduled post-operative admissions or patients on protocols that call for ICU-level monitoring) to the pediatric ICU (PICU) between January 2015 and June 2019 that met ET criteria – defined as having 1 or more of the following interventions within 1 hour before to 1 hour after transfer: intubation, vasopressor initiation, or ≥60 ml/kg fluid resuscitation.
This study was deemed exempt from review by our Institutional Review Board. Our use of standardized tools and language to identify cases with potential diagnostic delays and error were for the purposes of learning and continuous improvement, both locally and to contribute to the diagnostic medicine literature. These cases were reviewed by clinicians trained in analytic tools for possible opportunities in the diagnostic process, without consideration for any determination of a deviation in the standard of care.
Setting
The study took place at a large, free-standing children’s hospital with ~560 licensed inpatient beds, inclusive of ~300 medical/surgical beds and a 70-bed mixed medical-surgical PICU caring for >4,000 admissions annually.
The hospital utilizes a multifaceted rapid response system to recognize and respond to clinical deterioration events on non-cardiac medical and surgical wards. The system includes a high-risk patient identification “watcher” system that incorporates risk assessment using clinician judgement and an early warning score, a medical emergency team (MET) of critical care providers (ICU physician, nurse, and respiratory therapist) that responds within 30 minutes, and a code blue team that responds immediately.5,17 For the majority of the study period (January 2016–June 2019), the hospital used the pediatric Rothman Index18 as an early warning score and incorporated standardized escalation protocols for patients meeting threshold criteria.
During the study period, the cardiology ward had separate monitoring and escalation systems leading to transfer to a separate cardiac ICU. Similarly, escalations from the ward to the neonatal ICU had separate transfer and escalation systems. As such, patients with unplanned transfers in these clinical settings were not within the scope for our analysis.
Participants
We included all unplanned transfers from non-cardiac medical or surgical wards to the PICU between January 2015 and June 2019 that met ET criteria. Unplanned transfers to the pediatric ICU (e.g., not scheduled post-operative admissions or patients on protocols that call for ICU-level monitoring) were identified using the institution’s Virtual Pediatric Systems (VPS) database.19 When patients require escalation of care, >99% of patients transfer to the PICU. As discussed above, rarely, patients are escalated to the neonatal ICU or cardiac ICU; those (rare) transfers were not included in this study.
Only the first unplanned transfer during a hospital admission, the index transfer, was included. Patients with an active DNR at transfer were excluded as these patients may have differences in the extent of evaluating a new complaint and in management decisions regarding escalation of care.
ET criteria included having 1 or more of the following interventions within 1 hour before to 1 hour after transfer: intubation, vasopressor initiation, or ≥60 ml/kg fluid resuscitation. Unplanned transfers that met ET criteria were identified using an automated query of the institutional Enterprise Data Warehouse (EDW).3
Variables
Patient data including demographic information, originating service (the primary service caring for the patient prior to transfer to the ICU), hospital outcome data, and primary ICU diagnoses were obtained from the VPS database. Primary ICU diagnoses were grouped based on the VPS provided diagnosis category into one of four deterioration categories: circulatory, neurologic, respiratory, or other.3
Pertinent ET related intervention data and additional demographic characteristics which might impact risk of deterioration including the presence of a Complex Chronic Condition20 and Nationally Normed Child Opportunity Index Level21 were extracted from the EDW.3
Information about risk identification or assessment prior to ET including whether there was code blue activation at the time of the ET, any additional MET evaluation that occurred in the 48 prior to the ET (beyond the MET evaluation that occurred at time of ET), or recent transfer out of the PICU in the 48 hours prior to ET was obtained through chart review.
Case review to identify MOID and diagnostic process opportunities
We utilized existing screening and evaluation tools – revised Safer Dx questionnaire16,22 and modified Diagnostic Error Evaluation and Research (DEER) taxonomy10,23 – to identify and analyze MOIDs using clinical data and documentation in the EHR.
Reviewers
We convened a working group of 10 pediatric physician reviewers including attendings and fellows in critical care medicine, hospital medicine, and oncology. Reviewers were selected to provide balance in the group among clinical expertise and experience in quality improvement, patient safety, diagnostic excellence, and clinical deterioration prevention. At the study outset, institutional safety leaders with specialty training in diagnostic process improvement frameworks oriented the group to MOID and provided skills training for utilization of the SaferDx questionnaire. These leaders then facilitated collaborative reviews of example patient cases to ensure a shared understanding among the panel of MOID and the SaferDx questionnaire.
Identification of MOID using SaferDx
To provide an expansive perspective on potential opportunities for diagnostic improvement that incorporated both generalist and sub-specialist perspectives on disease course and diagnosis, each ET case was initially, independently screened by two reviewers, including an ICU physician paired with a physician in hospital medicine (for general medical/surgical patients) or in oncology (for patients admitted to the oncology service). Reviewers considered all EHR documentation, laboratory and radiology results, vital sign data, orders, and medication administration for a minimum of 48 hours prior to and following their escalation of care to the PICU.
Reviewers screened ETs using SaferDx,22 a 13-item instrument designed to identify the presence or absence of MOID within a patient care encounter. SaferDx has previously demonstrated reliability and validity in the identification of improvement opportunities in the pediatric acute care setting.16 Each item is scored on a Likert scale from 1 (strongly disagree) to 7 (strongly agree). The 13th item is a “global rating” based on the preceding 12 items using the same 1 through 7 scale. The final designation in item 13 indicates the potential presence or absence of a MOID based on the information available. Reviewers were instructed to evaluate cases for opportunities to improve diagnosis pertaining to both the reason for hospital admission and any new diagnosis related to the clinical deterioration event.
If either reviewer assigned a score ≥4 on the concluding (global rating) SaferDx question, indicating potential MOID, the case was discussed in detail with other members of the reviewer panel at a monthly meeting. After interdisciplinary discussion, the group determined a final global score. All ETs in which the final global score was ≥5 were classified as having MOID.
Evaluation of Diagnostic Process Improvement Opportunities in ETs with MOID
All ET cases classified as having MOID were evaluated with the modified DEER taxonomy tool10 in group discussion to identify specific diagnostic process improvement opportunities.10,23 The tool includes seven domains (each with multiple subdomains) that align with steps of the diagnostic process including Access to Care, History, Physical Exam, Testing, Hypothesis, Consultation, and Monitoring. DEER taxonomy allows for selection of all relevant domains/subdomains so that each case has the potential for multiple opportunities for diagnostic process improvement.
Statistical Analysis
We compared ET cases with and without MOID using descriptive statistics, including Fisher’s exact test for categorical variables. Multivariable regression was used to compare mortality and post-transfer LOS in days amongst cases with and without MOID with a priori defined confounders of age category, originating service, deterioration category, and pre-transfer LOS,3 which have been associated with risk of mortality in ET and were hypothesized to impact the diagnostic process in deteriorating patients.2,24 Mortality was modeled with logistic regression. We modeled post-transfer LOS with a generalized linear model with log link, gamma family, and robust variance estimates to account for the right skewed distribution.25,26 A p value less than 0.05 was considered significant. Statistical analyses were performed using STATA version 17.0 (Stata Corporation, College Station, TX).
Results
Description of the Cohort
Of 2,037 PICU transfers in the study period, 129 met criteria for ET (6.3%) (Figure 1). This cohort has previously been described in detail.3 Briefly, patients with ET had a median age of 7.4 years, 53% were non-white, and 38% had low or very low child opportunity index scores. Most patients had a complex chronic condition (97%) And were hospitalized for more than 24 hours prior to transfer (70%). ET cases most frequently originated from the Oncology service (43%) and involved circulatory deterioration (54%). Few ETs occurred following recent PICU transfer (7%) or had previous medical emergency team involvement (2%). More than a third included a code blue activation (36%).
Figure 1: Flowchart detailing breakdown of cohort for evaluation of missed opportunity to Improve Diagnosis (MOID).
Index Unplanned Transfer indicates first transfer from floor to ICU for that admission. Unplanned transfers were excluded if there was an active Do-Not-Resuscitate (DNR) order at the time of transfer. Unplanned transfers that met criteria for emergency transfer (ET) were included for evaluation of MOID.
ET = Emergency Transfer
ET criteria = patient received intubation, vasopressor initiation, or >= 60 ml/kg fluids within 1 hour before to 1 hour after ICU transfer.
MOID = Missed Opportunity to Improve Diagnosis as identified by case review process.
Incidence of MOID in ET
MOID was identified in 37 ETs (29%, 95% Confidence Interval (CI) 21%-37%). ET cases with MOID did not differ by age, sex, race, ethnicity, presence of complex chronic condition, child opportunity index, or deterioration category. ET cases originating from surgical services were more likely to have MOID, whereas those originating from oncologic services were less likely to have MOID – although the sample size was small (Table 1). Cases with MOID had similar rates of code blue activation at the time of their deterioration, prior MET evaluation, and recent transfer out of an ICU.
Table 1: Characteristics of the Cohort.
Comparison of patient characteristics between ET without MOID and ET with MOID. Data are presented as N (%). All p-values are the result of Fisher’s exact test.
| ET without MOID | ET with MOID | Total ET | p value | |
|---|---|---|---|---|
| Patients | 92 | 37 | 129 | |
| Age Category (y) | 0.71 | |||
| <1 | 11 (12) | 6 (16) | 17 (13) | |
| 1 to 4 | 25 (27) | 7 (19) | 32 (25) | |
| 5 to12 | 29 (32) | 11 (30) | 40 (31) | |
| >12 | 27 (29) | 13 (35) | 40 (31) | |
| Male | 50 (54) | 21 (57) | 71 (55) | 0.80 |
| Race | 0.21 | |||
| Asian | 7 (8) | 1 (3) | 8 (6) | |
| Black or African American | 17 (18) | 13 (35) | 30 (23) | |
| Multi-Racial | 1 (1) | 1 (3) | 2 (2) | |
| Other | 23 (25) | 6 (16) | 29 (22) | |
| White | 44 (48) | 16 (43) | 60 (47) | |
| Ethnicity | 0.61 | |||
| Hispanic or Latino | 17 (18) | 5 (14) | 22 (17) | |
| Not Hispanic or Latino | 75 (82) | 32 (86) | 107 (83) | |
| Nationally Normed Child Opportunity Index Score | 0.22 | |||
| Very High | 12 (13) | 7 (19) | 19 (15) | |
| High | 16 (17) | 8 (22) | 24 (19) | |
| Moderate | 11 (12) | 5 (14) | 16 (12) | |
| Low | 12 (13) | 1 (3) | 13 (10) | |
| Very Low | 23 (25) | 13 (35) | 36 (28) | |
| Unavailable | 18 (20) | 3 (8) | 21 (16) | |
| Complex Chronic Condition | 88 (96) | 37 (100) | 125 (97) | 0.20 |
| Originating Service | <0.01 | |||
| Oncology | 48 (52) | 8 (22) | 56 (43) | |
| Medical Subspecialty | 26 (28) | 13 (35) | 39 (30) | |
| General Pediatrics | 15 (16) | 8 (22) | 23 (18) | |
| Surgery | 3 (2) | 8 (22) | 11 (9) | |
| Deterioration category | 0.18 | |||
| Circulatory | 52 (57) | 17 (46) | 69 (54) | |
| Neurologic | 27 (29) | 10 (27) | 37 (29) | |
| Respiratory | 7 (8) | 8 (22) | 15 (12) | |
| Other | 6 (7) | 2 (5) | 8 (6) | |
| Pre-Transfer Length of Stay | 0.06 | |||
| <24 Hours | 31 (34) | 8 (22) | 39 (31) | |
| 24 Hours to 7 Days | 27 (29) | 20 (54) | 47 (36) | |
| 1 Week to 1 Month | 24 (26) | 8 (22) | 32 (25) | |
| >1 Month | 10 (11) | 1 (3) | 11 (9) | |
| Weekday daytime transfer | 37 (40) | 19 (51) | 56 (43) | 0.33 |
| Prior MET Activation | 1 (1) | 2 (5) | 3 (2) | 0.2 |
| Code Blue Activation | 34 (37) | 13 (35) | 47 (36) | 0.85 |
| Recent PICU Transfer | 5 (5) | 4 (11) | 9 (7) | 0.28 |
ET = Emergency Transfer
MOID = Missed Opportunity for Improvement in Diagnosis
Deterioration Category = Grouping of VPS designated primary ICU diagnosis
Prior MET Activation = Prior Medical Emergency Team evaluation that occurred in the 48 prior to the ET (beyond the MET evaluation that occurred at time of ET)
Recent PICU Transfer = Transfer out of the ICU within 48 hours prior to clinical deterioration event.
Weekday/daytime transfer = ICU transfers that occurred from 7:00AM to 6:59PM, Monday through Friday
Diagnostic Process Opportunities for Improvement
Applying the DEER Taxonomy tool to ET cases with MOID, we identified opportunities for improvement across all steps of the diagnostic process except for Access to Care (Table 2). Cases with MOID had a median of 4 identified diagnostic process steps with at least one opportunity for improvement. The most common opportunities identified involved Testing (33 cases 89%) and Physical Exam and Assessment (33 cases 89%) diagnostic process steps. Across all diagnostic process opportunities, the most frequently identified improvement opportunity was recognizing the urgency of an identified condition (31 cases, 84%). In over half of cases, we identified opportunities related to prioritization of data, ordering appropriate testing, and considering the correct diagnosis. Table 2 includes a brief vignette for each diagnostic process opportunity in which there were more than 5 cases identified, demonstrating the type of clinical event that was identified as an opportunity for improvement.
Table 2: Diagnostic Process Improvement Opportunities in ET Cases with MOID.
Classification of Emergency Transfers (ETs) with Missed Opportunities for Improved Diagnosis (MOID) using the Diagnostic Error Evaluation and Research project tool (DEER Taxonomy)10 to localize where opportunities existed in the diagnostic process. Example case vignettes provided for all opportunities with more than 5 cases. Diagnostic process improvement opportunities in which there were no cases are not included. Diagnostic process improvement opportunities and diagnostic process steps are not mutually exclusive, so a single case can have more than one diagnostic process improvement opportunity within each diagnostic process step.
| Diagnostic Process Step | Diagnostic Process Improvement Opportunity |
Number of Cases |
Example Case Vignette |
|---|---|---|---|
| History | Providing or eliciting a piece of history data | 3 | |
| Interpreting piece of history data | 2 | ||
| Weighing or prioritization of history data | 21 | A patient with a complex neurologic condition status post a neuro-vascular procedure with new neurologic changes that may have prompted earlier evaluation for stroke. | |
| Acting on or following-up on a piece of history data | 10 | A patient with immune suppression after an organ transplant, readmitted for nutrition and medication adjustments, had new vital sign changes that could have been evaluated in the context of a history of immune suppression and risk of infection. | |
| Physical Examination/Assessment | Perform a physical examination or assessment | 2 | |
| Physical examination or assessment finding | 7 | A patient awaiting an organ transplant developed oliguria, and tachycardia that may have been identified and assessed earlier as signs of potential shock. | |
| Weighing of a physical examination or assessment finding | 25 | A non-verbal patient admitted following a corrective surgical procedure in the abdomen with identified abdominal tenderness and oliguria that may have been weighed in context as signs of hemorrhagic shock. | |
| Acting on or following-up on a physical examination or assessment finding | 18 | A patient with medical complexity including chronic respiratory failure and recent history of acute respiratory distress syndrome developed progressive hypoxemia and work of breathing requiring escalation of respiratory support that may have led to an earlier evaluation for hypercarbia and new pulmonary infection. | |
| Testing (Laboratory/Radiology/Other) | Ordering needed test(s) | 21 | A baby with a recent history of minor traumatic head injury admitted with lethargy, poor feeding, and hypoglycemia that could have prompted additional testing for infection or intra-cranial pathology. |
| Performing needed test(s) | 3 | ||
| Test sequencing | 4 | ||
| Interpretation of test(s) | 9 | A patient with headache and fever had outside hospital images interpreted initially as brain mass, that upon repeat evaluation was interpreted as consistent with infection. | |
| Acting on or following-up on test result (including results not communicated to the patient | 4 | ||
| Hypothesis Generation | Considering correct diagnosis | 22 | A patient with a history of negative evaluation for a medical condition was admitted with a diagnosis of conversion disorder. Patient developed progressive symptoms that could have prompted re-consideration of a medical etiology and appropriate treatment. |
| Weighing or prioritizing | 16 | A patient with sickle cell disease admitted with pain crisis and new neurologic changes could have had evaluation of potential stroke prioritized given presenting symptoms and significance of the potential diagnosis. | |
| Too much weight given to lower probability or priority diagnosis | 13 | A patient receiving chemotherapy with fever and neutropenia was receiving broad-spectrum antibiotics but developed worsening abdominal tenderness, fever, and hypotension. Opportunity to focus diagnostic reasoning on higher priority intra-abdominal infection and potential septic shock over working diagnosis of unidentified infection. | |
| Referral/Consultation | Ordering a referral or consult | 2 | |
| Monitoring/Follow-up | Interpreting physiologic monitoring finding | 6 | A patient receiving palliative radiation who had a recent brain biopsy developed somnolence, change in ophthalmic exam, and evolving bradycardia and hypertension on cardiorespiratory monitoring that could have been interpreted as signs of progressive intra-cranial pathology rather than side-effects of opioid analgesia. |
| Recognizing urgency of condition or complication | 31 | A patient with a critical airway was admitted with viral upper respiratory infection, tachypnea, and progressively worsening oral secretions with an opportunity to recognize the urgency of new respiratory infection in the context of a known critical airway. | |
| Communicating findings among healthcare team members | 4 | ||
| Refer the patient to appropriate setting or for appropriate monitoring | 12 | A patient with hypoxia, tachycardia, and abdominal pain, found to have pneumatosis on abdominal CT with an opportunity to admit to an ICU rather than ward given the diagnosis. |
MOID Association with Outcomes
In a multivariable adjusted model, ET cases with MOID had significantly higher odds of mortality (odds ratio 5.5; 95%CI1.5 - 20.6; p 0.01) and longer post-transfer LOS (rate ratio 1.7, 95%CI 1.1-2.6, p 0.02) (Table 3).
Table 3: Association of ET with MOID and Primary Outcomes.
Primary outcome of ET cases with MOID compared to cases without MOID. Results presented are of a multivariable regression analysis adjusted for a-priori confounders: age category, originating service, deterioration category, and pre-transfer length of stay. Data presented as N (%) or median (interquartile range).
| Outcome | Total | With MOID | Without MOID | aOR/aRR | p value |
|---|---|---|---|---|---|
| N = 129 | N = 37 | N = 92 | |||
| In-Hospital Mortality | 17 (13) | 9 (24) | 8 (9) | 5.5 OR [95%CI 1.5 - 20.6] | 0.01 |
| Post-Transfer LOS | 14.7 (7.2-29.6) | 20.7 (7.2-36.7) | 12.4 (6.7-25.1) | 1.7 [95%CI 1.1 – 2.6] | 0.02 |
ET = Emergency Transfer
MOID = Missed Opportunity for Improvement in Diagnosis
LOS = Length of Stay
aRR = adjusted rate ratio
aOR = adjusted odds ratio
95%CI = 95% Confidence Interval
Discussion
We applied a diagnostic process improvement framework to assess MOID in clinical deterioration events. In a cohort of ETs, we found that MOID was common and associated with increased risk of in-hospital mortality and LOS. While frequency of MOID did not vary by demographic factors, pre-transfer LOS, or type of deterioration, a greater proportion of MOID occurred in ET cases originating from surgical services, although the sample size was small. The most frequently identified diagnostic process improvement opportunity was recognizing the urgency of an identified condition.
Our findings have important implications for both the clinical deterioration and diagnostic process literature. First, we note that the 29% MOID incidence rate is higher than has been previously estimated in other pediatric cohorts, which have estimated an incidence of MOID between 3.8% to 23% depending on the case source.13,14,16,27 ETs are an enriched cohort of clinical deterioration events for identifying MOID and, in contrast to the majority of pediatric inpatient studies of MOID13,14, encompass varied clinical contexts and patient diagnosis.
We found few differences in demographic factors or patient specific risk factors between ETs with and without MOID. Acknowledging the limitations of our sample size and our review process (which did not include a pediatric surgeon), surgical cases were infrequent in ETs overall but over-represented in cases with MOID. In contrast, oncologic cases were over-represented in ETs but had a relatively low proportion of MOID. This may suggest that there are novel challenges to the diagnostic process in settings where deterioration events are infrequent and that screening events in those setting may be high yield in identifying improvement opportunities.
Given the incidence of MOID and association with clinical outcomes in ETs, diagnostic process improvement frameworks could serve as an important adjunctive to existing efforts to prevent unrecognized clinical deterioration. Certainly, the increased risk of mortality in ETs with MOID suggests that diagnostic reasoning is central to our ability to identify and respond to life-threatening clinical changes. Most published efforts to prevent unrecognized clinical deterioration in hospitalized pediatric patients apply a situation awareness (SA) framework. SA—which refers simply to “knowing what’s going on”— has been conceptualized in terms of perception of information, comprehension in context, and projection to future states.28 SA is critical for identifying, mitigating, and escalating clinical deterioration, and interventions to improve SA have been shown to reduce ETs.29 The diagnostic process improvement tools applied here (SaferDx and DEER taxonomy) build on SA by specifying novel, actionable, systems improvement opportunities within SA domains.
Further definition of potential diagnostic process improvement opportunities is needed to design and test change ideas targeting diagnosis. The DEER taxonomy has been applied across clinical care environments to identify diagnostic process improvement opportunities.10,30-33 In ET cases with MOID, we found opportunities to improve weighing of relative importance of history, lab, and exam data elements and to support recognition of the urgency of diagnosis. While multi-center investigation is needed to evaluate generalizability of these findings, future clinical deterioration prevention efforts may benefit from systems to support prioritization of information that does not align with an existing working diagnosis or patient clinical course.
Our findings must be interpreted within the limitations of our approach. This is a single center study with potential for limited generalizability, given that nuanced differences may exist in criteria for ET and PICU admission across institutions. We evaluated only the first ICU transfer within an admission, potentially undercounting MOID in complex patients with prolonged hospitalization and multiple ICU transfers. This study also relied on retrospective review of clinical documentation—a design that is limited by accuracy and fidelity of documentation practices. We may have failed to capture diagnostic process steps incorporated into daily rounds or patient conversations but not entered into the medical record. Incorporating procedures to facilitate timely screening for MOID such as in clinical event debriefing systems or SA huddles could help to mitigate the potential biases resulting from retrospective chart review. Additionally, the diagnostic process tools used for this study, although widely utilized in diagnostic process improvement work, rely on subjective assessment of reviewers. While we assembled a diverse reviewer panel and instituted study procedures to ensure balanced perspectives were represented (e.g., pairing of ICU and non-ICU reviewers), there could be potential for bias. Though ours is the largest investigation of clinical deterioration to date, a larger number of events drawn from multiple institutions is needed to draw conclusions about the most important opportunities to improve diagnosis. Finally, our study period ends in 2019 and since that time there has been enhanced research and regulatory emphasis on improving diagnostic quality. It will be essential to evaluate the incidence and impact of MOID in a contemporary cohort of ETs to inform future improvement opportunities.
Conclusion
Applying a diagnostic process improvement framework to ETs, we found that MOID is common and associated with patient-centered outcomes including in-hospital mortality. Future multi-center investigation is needed to define specific opportunities for improving the diagnostic process to help promote early recognition and response to clinical deterioration.
Acknowledgements:
Dr. Sanjiv Mehta’s participation in this project was supported by the Pediatric Hospital Epidemiology and Outcomes Research Training (PHEOT) Program, an NICHD-funded postdoctoral fellowship (T32 HD060550). Dr. Ruiz’s effort was supported by the NIH Cancer Clinical Epidemiology Training Grant (T32-CA-009679) and the Abramson Cancer Center’s Paul Calabresi Career Development Award for Clinical Oncology (K12 CA076931). Dr. Rasooly’s effort was funded by K08 HS028682-01A1 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS). The authors are solely responsible for this document’s contents, findings, and conclusions, which do not necessarily represent the views of AHRQ, NICHD, or NIH. Readers should not interpret any statement in this report as an official position of AHRQ, NICHD, NIH, or of HHS. None of the authors has any affiliation or financial involvement that conflicts with the material presented in this report.
The authors thank the Center for Diagnostic Excellence at the Children’s Hospital of Philadelphia for their support of this work.
Footnotes
Conflict of Interest
The authors declare no conflict of interest.
Charles Phillips
Disclosures: Consultant for ChemoMap
Sanjiv D. Mehta, Morgan Congdon, Meghan Galligan, Christina M. Hanna, Naveen Muthu, Jenny Ruiz, Hannah Stinson, Kathy Shaw, Irit R. Rasooly
Disclosures: None
References
- 1.Sosa T, Galligan MM, Brady PW. Clinical progress note: Situation awareness for clinical deterioration in hospitalized children. Journal of Hospital Medicine. 2022;17(3):199–202. doi: 10.1002/jhm.2774 [DOI] [PubMed] [Google Scholar]
- 2.Hussain FS, Sosa T, Ambroggio L, Gallagher R, Brady PW. Emergency transfers: An important predictor of adverse outcomes in hospitalized children. Journal of Hospital Medicine. 2019;14(8):482–485. doi: 10.12788/jhm.3219 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mehta SD, Muthu N, Yehya N, et al. Leveraging EHR Data to Evaluate the Association of Late Recognition of Deterioration With Outcomes. Hospital Pediatrics. 2022;12(5):447–460. doi: 10.1542/hpeds.2021-006363 [DOI] [PubMed] [Google Scholar]
- 4.Brady PW, Muething S, Kotagal U, et al. Improving situation awareness to reduce unrecognized clinical deterioration and serious safety events. Pediatrics. 2013;131(1). doi: 10.1542/peds.2012-1364 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bonafide CP, Localio AR, Roberts KE, Nadkarni VM, Weirich CM, Keren R. Impact of Rapid Response System Implementation on Critical Deterioration Events in Children. JAMA Pediatrics. 2014;168(1):25. doi: 10.1001/jamapediatrics.2013.3266 [DOI] [PubMed] [Google Scholar]
- 6.Bonafide CP, Roland D, Brady PW. Rapid Response Systems 20 Years Later: New Approaches, Old Challenges. JAMA pediatrics. 2016;170(8):729–730. doi: 10.1001/jamapediatrics.2016.0398 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Difonzo M. Performance of the Afferent Limb of Rapid Response Systems in Managing Deteriorating Patients: A Systematic Review. Critical Care Research and Practice. 2019;2019. doi: 10.1155/2019/6902420 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Parshuram CS, Dryden-Palmer K, Farrell C, et al. Effect of a pediatric early warning system on all-cause mortality in Hospitalized pediatric patients: The epoch randomized clinical trial. JAMA - Journal of the American Medical Association. 2018;319(10):1002–1012. doi: 10.1001/jama.2018.0948 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Maharaj R, Raffaele I, Wendon J. Rapid response systems: a systematic review and meta-analysis. Crit Care. 2015;19(1):254. doi: 10.1186/s13054-015-0973-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Schiff GD, Kim S, Abrams R, et al. Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project. In: Henriksen K, Battles JB, Marks ES, Lewin DI, eds. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Advances in Patient Safety. Agency for Healthcare Research and Quality (US); 2005. Accessed October 23, 2019. http://www.ncbi.nlm.nih.gov/books/NBK20492/ [Google Scholar]
- 11.Henriksen K, Dymek C, Harrison MI, Brady PJ, Arnold SB. Challenges and opportunities from the Agency for Healthcare Research and Quality (AHRQ) research summit on improving diagnosis: a proceedings review. Diagnosis. 2017;4(2):57–66. doi: 10.1515/dx-2017-0016 [DOI] [PubMed] [Google Scholar]
- 12.Balogh EP, Miller BT, Ball JR, Error D, Care H, National T. Improving Diagnosis in Health Care. (Balogh EP, Miller BT, Ball JR, eds.). National Academies Press; 2015. doi: 10.17226/21794 [DOI] [PubMed] [Google Scholar]
- 13.Marshall TL, Rinke ML, Olson APJ, Brady PW. Diagnostic Error in Pediatrics: A Narrative Review. Pediatrics. 2022;149(Supplement 3):e2020045948D. doi: 10.1542/peds.2020-045948D [DOI] [PubMed] [Google Scholar]
- 14.Cifra CL, Custer JW, Singh H, Fackler JC. Diagnostic Errors in Pediatric Critical Care: A Systematic Review. Pediatr Crit Care Med. 2021;22(8):701–712. doi: 10.1097/PCC.0000000000002735 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Perrem LM, Fanshawe TR, Sharif F, Plüddemann A, O’Neill MB. A national physician survey of diagnostic error in paediatrics. Eur J Pediatr. 2016;175(10):1387–1392. doi: 10.1007/s00431-016-2772-0 [DOI] [PubMed] [Google Scholar]
- 16.Davalos MC, Samuels K, Meyer AND, et al. Finding Diagnostic Errors in Children Admitted to the PICU. Pediatric Critical Care Medicine. 2017;18(3):265–271. doi: 10.1097/PCC.0000000000001059 [DOI] [PubMed] [Google Scholar]
- 17.Winter MC, Kubis S, Bonafide CP. Beyond Reporting Early Warning Score Sensitivity: The Temporal Relationship and Clinical Relevance of "True Positive" Alerts that Precede Critical Deterioration. Journal of Hospital Medicine. 2019;14(3):138. doi: 10.12788/jhm.3066 [DOI] [PubMed] [Google Scholar]
- 18.Rothman MJ, Tepas JJ, Nowalk AJ, et al. Development and validation of a continuously age-adjusted measure of patient condition for hospitalized children using the electronic medical record. Journal of Biomedical Informatics. 2017;66:180–193. doi: 10.1016/J.JBI.2016.12.013 [DOI] [PubMed] [Google Scholar]
- 19.Bennett TD, Spaeder MC, Matos RI, et al. Existing Data Analysis in Pediatric Critical Care Research. Front Pediatr. 2014;2:79. doi: 10.3389/fped.2014.00079 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Feudtner C, Feinstein JA, Zhong W, Hall M, Dai D. Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation. BMC Pediatrics. 2014;14(1):199. doi: 10.1186/1471-2431-14-199 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Acevedo-Garcia D, McArdle N, Hardy EF, et al. The Child Opportunity Index: Improving Collaboration Between Community Development And Public Health. Health Affairs. 2014;33(11):1948–1957. doi: 10.1377/hlthaff.2014.0679 [DOI] [PubMed] [Google Scholar]
- 22.Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315–323. doi: 10.1515/dx-2019-0012 [DOI] [PubMed] [Google Scholar]
- 23.Schiff GD, Hasan O, Kim S, et al. Diagnostic Error in Medicine: Analysis of 583 Physician-Reported Errors. Archives of Internal Medicine. 2009;169(20):1881–1887. doi: 10.1001/archinternmed.2009.333 [DOI] [PubMed] [Google Scholar]
- 24.Mehta SD, Martin K, McGowan N, et al. Ventilator Weaning Pathway Associated with Decreased Ventilator Days in Pediatric ARDS. Crit Care Med. 2021;49(2):302–310. doi: 10.1097/CCM.0000000000004704 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Austin PC, Rothwell DM, Tu JV. A Comparison of Statistical Modeling Strategies for Analyzing Length of Stay after CABG Surgery. Health Services & Outcomes Research Methodology. 2002;3(2):107–133. doi: 10.1023/A:1024260023851 [DOI] [Google Scholar]
- 26.Verburg IWM, de Keizer NF, de Jonge E, Peek N. Comparison of Regression Methods for Modeling Intensive Care Length of Stay. PLOS ONE. 2014;9(10):e109684. doi: 10.1371/journal.pone.0109684 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Sawicki JG, Nystrom D, Purtell R, Good B, Chaulk D. Diagnostic error in the pediatric hospital: a narrative review. Hosp Pract (1995). 2021;49(sup1):437–444. doi: 10.1080/21548331.2021.2004040 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Endsley MR. Toward a theory of situation awareness in dynamic systems. Human Factors. 1995;37(1):32–65. [Google Scholar]
- 29.Sosa T, Sitterding M, Dewan M, et al. Optimizing Situation Awareness to Reduce Emergency Transfers in Hospitalized Children. Pediatrics. 2021;148(4):e2020034603. doi: 10.1542/peds.2020-034603 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Baartmans MC, Hooftman J, Zwaan L, van Schoten SM, Erwich JJHM, Wagner C. What Can We Learn From In-Depth Analysis of Human Errors Resulting in Diagnostic Errors in the Emergency Department: An Analysis of Serious Adverse Event Reports. J Patient Saf. 2022;Publish Ahead of Print. doi: 10.1097/PTS.0000000000001007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Balogh Erin P., Miller Bryan T., and Ball John R. EC, Services; on DE in HCB on HC, Sciences I of MTNA of, Engineering and M. Improving Diagnosis in Health Care. [Google Scholar]
- 32.Rasooly IR, Dang K, Nawab US, Shaw KN, Wood JN. Applying a diagnostic excellence framework to assess opportunities to improve recognition of child physical abuse. Diagnosis (Berl). 2022;9(3):352–358. doi: 10.1515/dx-2022-0008 [DOI] [PubMed] [Google Scholar]
- 33.Schiff GD. Diagnostic Error in Medicine: Analysis of 583 Physician-Reported Errors. Archives of Internal Medicine. 2009;169(20):1881. doi: 10.1001/archinternmed.2009.333 [DOI] [PubMed] [Google Scholar]