Abstract
Background/Objective:
We evaluated the feasibility and discriminability of recently proposed Clinical Performance Measures for Neurocritical Care (Neurocritical Care Society) and Quality Indicators for Traumatic Brain Injury (CENTER-TBI) extracted from electronic health record (EHR) flowsheet data.
Methods:
At 3 centers within the Collaborative Hospital Repository Uniting Standards for Equitable AI (CHoRUS) consortium, we examined consecutive neurocritical care admissions exceeding 24 hours (03/2015–02/2020), and evaluated the feasibility, discriminability, and site-specific variation of five clinical performance measures and quality indicators: 1) ICP monitoring (ICPM) within 24 hours when indicated, 2) ICPM latency when initiated within 24 hours, 3) frequency of nurse-documented neurologic assessments, 4) intermittent pneumatic compression device (IPCd) initiation within 24 hours, and 5) latency to IPCd application. We additionally explored associations between delayed IPCd initiation and ICD-10-documented venous thromboembolism (VTE). Median [IQR] statistics are reported. Kruskal-Wallis tests were measured for differences across centers, and Dunn statistics were reported for between-center differences.
Results:
14,985 admissions met inclusion criteria. ICPM was documented in 1,514 (10.1%), neurologic assessments in 14,635 (91.1%), and IPCd application in 14,175 (88.5%). ICPM began within 24 hours for 1,267 (83.7%) with site-specific latency differences among sites 1–3, respectively, (0.54h [2.82], 0.58h [1.68], and 2.36h [4.60]; p<0.001). The frequency of nurse-documented neurologic assessments also varied by site (17.4/day [5.97], 8.4/day [3.12], and 15.3/day [8.34]; p<0.001) and diurnally (6.90/day during daytime hours vs. 5.67/day at night, p < 0.001). IPCd were applied within 24 hours for 12,863 (90.7%) of patients meeting clinical eligibility (excluding those with EHR documentation of limiting injuries, actively documented as ambulating, or refusing prophylaxis). In-hospital VTE varied by site (1.23%, 1.55%, and 5.18%; p<0.001) and was associated with increased IPCd latency (Overall. 1.02h [10.4] vs. 0.97h [5.98], p = 0.479; Site 1: 2.25h [10.27] vs. 1.82h [7.39], p=0.713; Site 2: 1.38h [5.90] vs. 0.80h [0.53], p=0.216; Site 3: 0.40h [16.3] vs. 0.35h [11.5], p=0.036).
Conclusions:
EHR-derived reporting of neurocritical care performance measures is feasible and demonstrates site-specific variation. Future efforts should examine whether performance or documentation drives these measures, what outcomes are associated with performance, and whether EHR-derived measures of performance measures and quality indicators are modifiable.
Keywords: Quality Indicators, Health Care, Quality Improvement, Electronic Health Records, Data Science, Intracranial Pressure, Venous Thrombosis, Glasgow Coma Scale, Big Data, Critical Care
Introduction
Identification of clinical performance measures and quality indicators for critical care informs guidelines, allows for objective comparisons of practice-pattern variability, and serves as incentive for improving the quality of clinical care and clinical outcomes1. Performance measures broadly fall into three domains: structure, process, and outcome2,3. Structure indicators are derived from hospital-level impact on care. These may include clinical standards, quality assurance audits, and availability of staff, equipment, or space1. Process indicators generally capture treatment directly applied to patients, such as time from injury to admission, number of patients meeting nutritional goals, or number of patients receiving pharmaceutical prophylaxis. Outcome indicators evaluate the disposition and recovery of the patient population following a health event of interest. Derived from each of these categories, quality measures can provide valuable insights into the performance of care and targets for improvement.
Potential performance measures can be evaluated for inclusion in guidelines on several criteria including feasibility, discriminability, and actionability4. Feasibility refers to the potential for objective evaluation, including data availability. Discriminability focuses on whether performance is variable enough across patients, providers, or sites, such that meaningful differences can be measured. Actionability refers to measures that lend themselves to tangible and realistic interventions to improve care.
Examples of highly successful quality indicator projects include the American Heart Association’s Get with the Guidelines for Stroke and Cardiovascular Disease5,6. These programs have provided evidence and support for the benefits of quality measurement6,7. However, a specific set of guidelines for neurocritical care and/or other disease conditions have yet to be collected, and current practice guidelines can vary widely between providers, centers, and countries8,9. Several efforts have proposed clinical performance measures and quality indicators in the realm of neurocritical care. The Neurocritical Care Society’s Guidelines Committee proposed 21 performance measures that were developed across nine different conditions within the broad realm of neurocritical care by analysis of citations in the broader literature10. Additionally, the CENTER-TBI consortium derived quality indicators for traumatic brain injury (TBI) management through a Delphi process involving clinical experts2. They evaluated data collected in the CENTER-TBI cohort for validity, feasibility, discriminability and actionability, proposing 26 structure and process measures3. However, proposed measures have not yet been evaluated using pragmatic methods to collect data at scale without costly efforts of detailed chart review.
We sought to evaluate an approach using granular structured electronic health record (EHR) data to enhance the scalability, feasibility, and validity of quality indicators and performance measures across centers. We selected specific measures identified by the CENTER-TBI and NCS groups with the aim of validating Neurocritical Care Unit (NCCU)-specific quality indicators with automated electronic health record data: availability of ICP monitoring (ICPM) (structure), time to ICPM (process), frequency of documented neurologic assessments (process), use of intermittent pneumatic compression devices (IPCd) to prevent deep vein thrombosis (DVT) (process), and time to placement of IPCds (process).
Methods
Patient Cohort
We developed a retrospective multi-center cohort consisting of consecutive admissions between March 2015 and February 2020 at three academic medical centers’ dedicated NCCUs within the Collaborative Hospital Repository Uniting Standards (CHoRUS) consortium – an open critical care workgroup with goals to facilitate the training, ethical guidance, standards, data acquisition architecture, and analytical tools to enable multi-center and multimodal critical care research. The consortium includes members from over 20 academic medical centers with representation from surgical critical care, pulmonary critical care medicine, and neurocritical care. We excluded admissions of less than 24 hours and those for which flowsheet data was unavailable. Data was captured from institutional electronic data warehouses, included clinical and demographic findings, and was collected under Institutional Review Board-approved protocols.
Selection of Candidate Measures
Candidate measures were selected from those previously defined by the CENTER-TBI consortium and the NCS Guidelines Committee, targeting their overlap (Figure 2). Measures hypothesized to have high feasibility and discriminability were selected for their potential to be validated using electronic health record.
Figure 2.
Diagram of project program. PM were selected from NCS and CENTER-TBI proposals for field-testing and evaluated on EHR databases from three sites, and include ICPM within 24 hours when indicated, time to ICPM, mechanical DVT prophylaxis within 24 hours, time to mechanical prophylaxis, and frequency of nurse-documented GCS assessments. ADT and diagnostic data supplemented and enabled analysis.
Timing of intracranial pressure (ICP) monitoring:
CENTER-TBI recommendations proposed ICPM as both a structure indicator and a process indicator: specifically, availability and frequency of use when clinically indicated. Similarly, the NCS Guidelines Committee metrics included elevated intracranial pressure and intracranial hypertension as an important disease condition, although concluded that insufficient data precluded implementing the quality measure. We hypothesized that the granular and frequent nature of ICP measurements would be suitable for an EHR-extraction approach. We therefore pre-specified two measures of ICPM: acute monitor placement (defined as that placed within 24 hours), and the time from NCCU admission to ICPM placement within those conditions. The first record of an ICP measurement was used as the reference timestamp.
Frequency of nurse-documented neurologic assessments:
CENTER-TBI recommendations proposed the frequency of assessment scales in the NCCU as a quality indicator, specifically Glasgow Coma Scale (GCS) motor scores, but lacked data to validate it within their cohort. We collected all GCS assessment data during the NCCU admission and derived the ratio of the number of assessments (total GCS) to the total number of ICU days. These times include potential periods where patients may have been unavailable for GCS scoring (time outside NCCU, under anesthesia, etc.). Assessments conducted while under sedation or paralysis were included in frequency calculations, as the assessment score (which is potentially affected by bias in these cases) is not included in this analysis.
Latency of intermittent pneumatic compression device (IPCd) placement:
Both the CENTER-TBI group and the NCS clinical performance measures highlighted venous thromboembolism (VTE) prophylaxis as a key quality indicator. The NCS clinical performance measure was defined as the percentage of patients who developed VTE but did not receive appropriate prophylaxis. The CENTER-TBI group proposed both chemical and mechanical prophylaxis, validating the former but not achieving a sufficiently high feasibility (percent available data) to be able to endorse the latter’s use. They calculated the percentage of patients for whom mechanical DVT prophylaxis was initiated within 24 hours of NCCU admission, of the patients eligible. We preferentially selected the CENTER-TBI measure, hypothesizing that using readily extracted, automatable EHR data would improve feasibility. We evaluated performance of the CENTER-TBI group’s definition of mechanical prophylaxis, but additionally explored associations between delayed IPCd application and ICD-10-documented in-hospital VTE. Documented VTE was chosen in lieu of documented DVT or lower extremity DVT to reduce potential bias introduced by documentation differences between sites, with the umbrella diagnosis identifying a more complete set of patients. Both overall frequency of IPCd placement and latency per patient were examined. Sufficient mechanical DVT prophylaxis was defined as any mechanism of mechanical prophylaxis applied to at least one extremity. Patients who had structured EHR documentation of refusing prophylaxis, having exclusionary injuries, or who were ambulating within the first 24 hours were excluded.
Data Collection and Processing
Data was extracted from local electronic data warehouses at the respective CHoRUS sites. Deidentified data was shared between sites under existing data use agreements. Data collected included NCCU Admission-discharge-transfer (ADT) records, clinical flowsheet data, and diagnostic codes. SQL language was used for data queries and for isolating relevant datapoints of interest in Microsoft SQL Server Management Studio v18.9.1 and BigQuery in the Google Cloud Platform Console.
ADT records were filtered by hospital unit and complied per patient encounter. Encounter start and end times were defined by earliest entrance and latest exit from NCCU, regardless of disposition. This included transfers in and out, admission, and discharge events. Encounter start and end times were used to derive the length of NCCU stay (LOS). Admission timing (weekday vs. weekend and daytime vs. nighttime) were also hard coded and considered for analysis.
Flowsheet data was selected for all three chosen measures: ICPM, neurologic assessments, and IPCd application. A dictionary of relevant flowsheet variable names was developed for each site, specific to their EHR configuration (Appendix A). Flowsheet data within each encounter was filtered to include only those data points recorded during NCCU stay, based on ADT encounter start and end times. Elapsed times from encounter start to individual data point records were also calculated. GCS scores were further coded for record timing (daytime vs. nighttime). IPCd data was filtered for exclusion criteria (notes that patient was ambulating or refused application, etc). IPCd and ICPM data were further filtered to include only records with elapsed time less than 24 hours, then the first record per patient encounter.
Diagnostic code data was filtered to include records added prior to encounter end-time. Data was categorized into disease conditions by ICD-10 code grouped as per a custom dictionary intended to reflect practice patterns (Appendix B). Encounters were eligible for inclusion in more than one category if more than one ICD-10 code was applied. Encounters with ICD-10 code(s) available but no matching disease conditions were categorized as “other.” A small number of encounters lacked diagnostic data altogether; these were categorized as “undocumented.” An additional flag was applied to diagnostic code records that were added after NCCU encounter start and matched a ICD-10 dictionary for VTE as previously used11, for use in IPCd measure evaluation.
Data Validation
Manual chart review was conducted for a sample of encounters following data collection and processing, to ensure integrity of data following extraction from data warehouses. Minor errors were identified, which allowed for correction of methodology that was then applied to the cohort as a whole. Such errors included inaccurate attributions of encounter start and end times, resulting from mismatched data warehouse “effective” vs. “enacted” date stamps. Rectifying these errors eliminated discrepancies between the EHR and EDW-derived data cohort. No major errors were found. Further validation of EHR data using in-person observations were not possible, given the retrospective nature of the study, and the goals of performance measures to encourage documentation for reasons not achieving a performance measure12,13 and to assess the value of contextual annotations, for example reason IPCd not applied as above.
Statistical Analysis
We specified null hypotheses that no significant differences would exist between sites for ICP latency, neurologic assessment frequency, and VTE diagnosis. Additionally, we specified a null hypothesis that no significant differences in IPCd latency would exist between patients with in-hospital VTE diagnoses and those without, within each site. We examined these associations using the Kruskal-Wallis one-way analysis of variance test for non-parametric data. Comparisons between individual pairs of sites were examined using Dunn’s test for multiple comparisons with Bonferroni correction. Further multivariate analyses addressed potential confounding.
Results
Patient Characteristics
14,895 admission encounters met criteria for inclusion (Site 1: 5906, Site 2: 5340, Site 3: 3739; Figure 1). Of these, the median age was 62 years (interquartile range [IQR], 49–73; Table 1). 3815 encounters were weekend admissions (23.7%), while 6599 were overnight admissions (40.9%). The median NCCU duration of stay was 2.67 days (IQR, 1.64–5.49), with 9280 (61.9%) of admissions exceeding 48 hours. Encounters were categorized by disease condition (Table 1): the most common were cerebral ischemia (4595, 30.7%); spontaneous intracerebral hemorrhage (2308, 15.4%); traumatic brain injury and nontraumatic subdural hematoma (1895, 12.6%); nontraumatic subarachnoid hemorrhage (1688, 11.3%); benign (non-cancerous) head and neck tumors including spine tumors (1619, 10.8%); and seizure disorders (1189, 7.9%). 1775 (11.0%) other encounters did not fall into a pre-defined category, which likely represent medical or surgical overflow (conditions such as sepsis, ARDS, etc.). 600 (3.7%) encounters did not have ICD-10 codes.
Figure 1.
CONSORT (Consolidated Standards of Reporting Trials) diagram. ICU indicates intensive care unit; ICPM, intracranial pressure monitoring; and IPCd, intermittent pneumatic compression devices.
Table 1.
Patient Characteristics
| Site 1 n=5,906 | Site 2 n=5,340 | Site 3 n=3,739 | Overall N=14,985 | |
|---|---|---|---|---|
|
| ||||
| Demographics | ||||
| Age, median (IQR) | 63 (50 – 73) | 61 (49 – 73) | 61 (49 – 73) | 62 (49 – 73) |
| Admission timing | ||||
| Weekend, n (%) | 1288 (21.8%) | 1164 (21.8%) | 1041 (28.0%) | 3815 (23.7%) |
| Overnight, n (%) | 1914 (32.4%) | 1772 (33.2%) | 1535 (41.1%) | 5221 (34.8%) |
| Length of NCCU stay [LOS] greater than 48 hours, n (%) | 3304 (55.9%) | 3442 (64.5%) | 2534 (67.8%) | 9280 (61.9%) |
| LOS (days), median (IQR) | 2.27 (1.39 – 4.98) | 2.80 (1.76 – 5.64) | 2.99 (1.76 – 6.68) | 2.71 (1.65 – 5.65) |
| Disease subgroup, n (%) | ||||
| Cerebral ischemia | 1386 (23.5%) | 1090 (20.4%) | 2119 (56.7%) | 4595 (30.7) |
| Nontraumatic subarachnoid hemorrhage | 612 (10.4%) | 475 (8.9%) | 601 (16.1%) | 1688 (11.3%) |
| Spontaneous intracerebral hemorrhage | 808 (13.7%) | 658 (12.3%) | 842 (22.5%) | 2308 (15.4%) |
| Traumatic brain injury and Nontraumatic subdural hematoma | 600 (10.2%) | 553 (10.4%) | 742 (19.8%) | 1895 (12.6%) |
| Structural or Dynamic CSF disorders | 131 (2.2%) | 80 (1.5%) | 460 (12.3%) | 671 (4.5%) |
| Malignant brain tumors | 400 (6.8%) | 277 (5.2%) | 119 (3.2%) | 796 (5.3%) |
| Other head and neck tumors, including benign brain/spine tumors | 650 (11.0%) | 751 (14.1%) | 218 (5.8%) | 1619 (10.8%) |
| Seizure disorders | 494 (8.4%) | 510 (9.6%) | 185 (4.9%) | 1189 (7.9%) |
| Inflammatory, infectious, and auto-immune cerebral disorders | 131 (2.2%) | 103 (1.9%) | 335 (9.0%) | 569 (3.8%) |
| Cardiac arrest, anaphylaxis, and risk of anaphylaxis | 60 (1.0%) | 61 (1.1%) | 234 (6.3%) | 355 (2.4%) |
| Neuromuscular disorders | 38 (0.6%) | 84 (1.6%) | 176 (4.7%) | 298 (2.0%) |
| Spinal cord injury, tumor, and infections | 120 (2.0%) | 84 (1.6%) | 378 (10.1%) | 582 (3.9%) |
| Vasculopathy | 8 (0.1%) | 8 (0.1%) | 56 (1.5%) | 72 (0.5%) |
| Other | 1052 (17.8%) | 648 (12.1%) | 57 (1.5%) | 1757 (11.7%) |
| Undocumented | 126 (2.1%) | 473 (8.9%) | 1 (0.03%) | 600 (3.7%) |
Feasibility
Across all sites, 14,635 encounters contained data for at least one GCS assessment in the NCCU (91.4%, Table 2). 14,175 contained eligible data entry for IPCd (88.5%). These results all exceed the 70% threshold the CENTER-TBI group set for data availability. 1514 encounters contained at least one ICPM measurement while in the NCCU (10.1%), consistent with the expected ratio based on clinical indication. Stratification of the data by maximum GCS score through the first 24 hours of NCCU admission supports the data’s availability, as the number receiving ICPM was comparable to the number of encounters with GCS less than 9.
Table 2.
Performance Measure Feasibility
| Site 1 | Site 2 | Site 3 | Overall | |
|---|---|---|---|---|
|
| ||||
| ICPM | ||||
| n | 499 | 696 | 319 | 1514 |
| Max GCS < 9 within 24 h | 499 | 424 | 452 | 1375 |
| Monitoring within 24 h, n (%) | 420 (84.2%) | 608 (87.4%) | 239 (74.9%) | 1267 (83.7%) |
|
| ||||
| Mechanical prophylaxis | ||||
| n | 5782 | 5281 | 3112 | 14175 |
| % available data | 97.9% | 98.9% | 83.2% | 88.5% |
| Application within 24 hr [n (%)] | 5394 (93.3%) | 5123 (97.0%) | 2345 (75.4%) | 12862 (90.7%) |
| Patients Documented with VTE | ||||
| VTE (n) | 56 | 71 | 128 | 255 |
| No VTE (n) | 4394 | 4540 | 2334 | 11268 |
| Percent VTE | 1.2% | 1.5% | 5.2% | 2.2% |
|
| ||||
| Neurologic assessments | ||||
| n | 5865 | 5337 | 3433 | 14635 |
| % available data | 99.3% | 99.9% | 91.8% | 91.4% |
Discriminability
Of the 1514 encounters with ICPM, monitoring was initiated within 24 hours of NCCU admission for 1267 (83.7%; Site 1: 420, 84.2%; Site 2: 608, 87.4%; Site 3: 239, 74.9%; Table 2). These 1267 encounters, representing cases with ICPM indicated during the acute phase, had differences in the time to first ICP measurement between sites. The median time at Site 3 (2.36 hours [IQR, 0.77–5.38]) was significantly different than at Site 1 (0.54 hours [IQR, 0–2.82]; p < 0.001) and Site 2 (0.58 hours [IQR, 0.23–1.92]; p < 0.001), although differences between Site 1 and Site 2 were not significant (p = 0.408). Multivariate analysis incorporating maximum GCS score through the first 24 hours after NCCU admission found no interaction. Further multivariate analyses found no differences by year, with site-specific difference preserved (2016: Site 1 vs. Site 2, p = 0.175; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001; 2017: Site 1 vs. Site 2, p = 1; Site 1 vs Site 3, p = 0.043; Site 2 vs. Site 3, p = 0.004; 2018: Site 1 vs Site 2, p = 0.008; Site 1 vs. Site 3, p < 0.001; Site 2 vs Site 3, p < 0.001; 2019: Site 1 vs. Site 2, p = 1; Site 1 vs. Site 3, p = 0.003; Site 2 vs. Site 3, p < 0.001; Figure 3A). 2019 data included January and February of 2020. Significant differences were seen overall by diagnosis, categorized by SAH, TBI, both, and neither, with TBI patients receiving monitoring later than SAH patients (1.44 hours [IQR, 0.47–5.53] vs. 0.64 hours [IQR, 0.06–2.78], p = 0.002). These differences were also significant at Site 2 but not at Sites 1 and 3 (Site 1: 0.35 [IQR, −23.7–1.72] vs. 0.55 [IQR, −1.42–3.00], p = 0.953; Site 2: 1.242 [IQR, 0.40–3.01] vs. 0.42 [IQR, 0.03–1.95], p = 0.002; Site 3: 3.48 [IQR, 0.96–8.07] vs. 1.93 [IQR, 0.68–4.28], p = 0.186; Figure 3D).
Figure 3.
Analysis of PM metrics over time and by patient diagnosis. Medians and 95% confidence intervals are represented. (A-C) Hours from NCCU admission to first ICP measurement, numbers of GCS assessments documented per admission day, and hours from NCCU admission to documentation of mechanical prophylaxis are evaluated over year of admission. *2019 includes data through February 2020. Between site differences are observed in all three measures across time. (D-F) Hours from NCCU admission to first ICP measurement, numbers of GCS assessments documented per admission day, and hours from NCC admission to documentation of mechanical prophylaxis are evaluated by diagnosis. CI indicates cerebral ischemia and ICH, intracranial hemorrhage. Years with documentation greater 65% are plotted. (D) ICPM is refined by indicating diagnoses (subarachnoid hemorrhage [SAH] and traumatic brain injury [TBI]). Negative values represent ICPM initiation prior to NCCU admission. Between site differences are observed within neither, SAH, and TBI groups. Further differences exist between diagnoses at Sites 2 and 3. (E) Between site differences in frequency of GCS assessment are preserved across diagnosis. (F) Diagnosis affects time to IPCd across sites.
Of the 14175 encounters with eligible IPCd data, IPCd initiation was documented within 24 hours for 12682 (90.7%; Site 1: 5394, 93.3%; Site 2: 5123, 97.0%; Site 3: 2345, 75.4%; Table 2). These 12682 encounters were stratified by development of in-hospital VTE after NCCU admission, using ICD-10 codes, and length of NCCU stay (LOS, above or below 48 hours). 255 cases of in-hospital VTE were documented overall (2.2%; Site 1: 56, 1.2%; Site 2: 71, 1.5%; Site 3: 128, 5.2%). Additionally, multivariate analyses examined time trends and disease specific differences by evaluating the effects of admission year and principal diagnosis on the median time to documentation of IPCd placement. A time trend was evident with documentation of IPCd initiation earlier in successive years (Figure 3C). However, this did not account for the significant differences between sites (2016: Site 1 vs. Site 2, p = 0.006; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001; 2017: Site 1 vs. Site 2, p = 0.002; Site 1 vs Site 3, p < 0.001; Site 2 vs. Site 3, p = 0.003; 2018: Site 1 vs Site 2, p < 0.001; Site 1 vs. Site 3, p < 0.001; Site 2 vs Site 3, p = 0.106; 2019: Site 1 vs. Site 2, p < 0.001; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001). In contrast, the median time to documentation of placement varied according to the principal diagnosis (cardiac diagnosis, p = 0.043; CI. p < 0.001; CSF dynamics diagnosis, p < 0.001; ICH, p = 0.006; Inflammation, p < 0.001; Malignant tumors, p < 0.001; Neuromuscular disorders, p = 0.002; Other, p < 0.001; Other tumors, p < 0.001; SAH, p < 0.001; Seizures, p = 0.056; Spinal cord diagnosis, p < 0.001; TBI, p = 0.040; Vasculopathy, p = 0.061; Figure 3F).
The overall median time to documentation of placement was higher in the VTE group than the non-VTE group, but was not statistically significant (1.02 hours [IQR, 0.22–10.8] vs. 0.97 hours [IQR, 0.28–6.26], p = 0.479). Although this trend also existed at individual sites, significant differences only existed at Site 3 (Site 1: 2.25 hours [IQR, 0.53–10.9] vs. 1.82 hours [IQR, 0.433–7.82], p = 0.713; Site 2: 1.38 hours [IQR, 0.44–6.34] vs. 0.80 hours [IQR, 0.32–3.73], p = 0.216; Site 3: 0.40 hours [IQR, 0.133–16.4] vs. 0.35 hours [IQR, 0.12–11.6], p = 0.036). Time-to-IPCd placement documentation was most rapid at Site 3 for both patients with or without a VTE diagnosis. The median time to documentation of IPCd placement was significantly later overall in the subpopulation of patients requiring a LOS greater than 48 hours (1.32 hours [IQR, 0.32–7.65] vs. 0.68 hours [IQR, 0.23–4.32], p < 0.001). While this trend was demonstrated across sites, the differences were only significant at Site 1 and Site 3 (Site 1: 2.78 hours [IQR, 0.60–9.35] vs. 1.03 hours [IQR, 0.32–5.73], p < 0.001; Site 2: 0.95 hours [IQR, 0.35–4.33] vs. 0.60 hours [IQR, 0.27–3.08], p = 0.055; Site 3: 0.52 hours [IQR, 0.13–21.1] vs. 0.20 hours [IQR, 0.08–3.6], p < 0.001). Stratifying by site reduced the association between documentation of IPCd initiation and VTE diagnosis but revealed one site with a significant association between delayed IPCd initiation and VTE among patients with LOS exceeding 48 hours (Figure 4), and in the overall population among patients with LOS exceeing 48 hours group (1.36 hours [IQR, 0.32–7.6] to IPCd initiation among patients with LOS exceeding 48 hours with VTE vs. 1.32 hours [IQR, 0.23–11.2] among patients with LOS exceeding 48 hours without VTE, p < 0.001) (Figure 4).
Figure 4.
Effect of documentation of in-hospital venous thromboembolism (VTE) on mechanical prophylaxis performance. Between site differences statistically significant in the absence of VTE. Further significant difference is observed between presence of VTE. Medians and 95% confidence intervals are represented. ***p < 0.005. Statistical comparisons were carried out using Kruskal-Wallis one-way analysis of variance test for non-parametric data and Dunn’s test for multiple comparisons with Bonferroni correction.
For the 14,635 encounters with nurse-documented GCS flowsheet data, Site 1 had the greatest median frequency of GCS measure documentation per 24-hour period (17.4 [IQR, 14.14–20.11]), followed by Site 3 (15.3 [IQR, 10.61–18.96]), then Site 2 (8.4 [IQR, 6.94–10.01]). Pairwise comparisons demonstrated significant differences between all three sites (Site 1 vs. Site 2, p < 0.001; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001). Stratification by initial GCS score and disease condition yielded no interaction despite significant disease-specific differences (Figure 3E). Additionally, multivariate analysis for time trends demonstrated that the association between site and frequency of GCS score assessments was independent of admission year (p < 0.001), despite some within-site variation over time (Figure 3B). There was a significant interaction between frequency of GCS assessment and the time of assessment (daytime [8am to 8pm] vs. nighttime) (p < 0.001). Median GCS frequency was consistently significantly greater during the daytime than the nighttime across all sites (Overall: 6.90 [IQR, 4.80–9.84] vs. 5.67 [IQR, 3.53–8.58], p < 0.001; Site 1: 9.38 [IQR, 7.22–11.1] vs. 8.18 [IQR, 6.92–9.61], p < 0.001; Site 2: 4.95 [IQR, 4.02–5.95] vs. 3.49 [IQR, 2.68–4.29], p < 0.001; Site 3: 8.12 [IQR, 5.43–10.2] vs. 7.2 [IQR, 5.08–9.10], p < 0.001; Figure 5A). Assessment schedule varied within sites by hour of the day (Figure 5B). However, site-specific differences were independent of these diurnal trends (Daytime: Site 1 vs. Site 2, p < 0.001; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001; Nighttime: Site 1 vs. Site 2, p < 0.001; Site 1 vs. Site 3, p < 0.001; Site 2 vs. Site 3, p < 0.001).
Figure 5.
Diurnal trends in GCS assessment timing. Medians and 95% confidence intervals are represented. (A) GCS assessment frequency is greater during daytime hours (8am to 8pm) than nighttime hours across sites. (B) GCS assessments per hour varies periodically varies by hour, with greatest frequency every four hours. Fluctuations occur within sites.
Discussion
For recommended clinical performance measures and quality indicators,14,15 an at-scale data science approach with EHR extraction of data and contextual information was feasible for intracranial pressure monitoring latency, frequency of neurologic assessments, and mechanical VTE prophylaxis. High availability of data allowed for analysis of the chosen measures and suggests that current documentation practices are sufficient to enable wide-spread use of this tool. Access to electronic data warehouses at the included sites was a requirement for this approach, as was selection of measures for which the numerator and denominator could be reasonably captured from existing documentation, rather than from clinician interviews. This approach specifically enabled feasibility for quantifying mechanical DVT prophylaxis, measuring neurologic assessment frequency, and ICPM timing, where these measures were previously considered to have uncertain feasibility2,3.
Site-specific variation was evident in all three measures evaluated, suggesting the measures are potentially good candidates for quality indicators, as the site-specific variation indicates existing procedural and/or performance differences between sites and commensurate opportunities for quality improvement. Also identified were diurnal trends in neurologic assessments, occurring less frequently at night. Future directions should likely include correlating measures to differential patient outcomes to assess if they should be recommended for inclusion in future guidelines. Early indications in this study support that one measure, IPCd application, may be a good candidate to meet the actionability criteria, given its association with new in-hospital VTE during admissions longer than 48 hours. However, these results require clinical adjudication of VTE diagnosis, and further efforts to control for baseline patient severity, differences in documentation practices, and differences in the rate of diagnostic studies. For example, patients without IPCd may undergo higher rates of VTE surveillance, leading to higher rates of detection16,17.
In addition, with increasing efforts to derive insights from regional and national electronic medical record data sources, identifying performance measures and quality indicators may be useful in calibrating machine learning models to maximize relevance at each local site. Indeed, prior work has shown that differences in these quality measures can impact the fidelity of models that generalize across geographic sites18,19. The ability to characterize domain shifts, and thereby correct for such instances may allow for improved and more generalized learning algorithms to be deployed across neurocritical ICUs.
The primary limitation of this study is the likelihood of site-specific variation in documentation and EHR usage. Site-specific usage of ICD-10 codes and differences in VTE surveillance may affect diagnosis frequency and VTE categorization within our cohort, as increased surveillance has been previously shown to increase detection and therefore documentation17,20, but may not represent actual increased instance of disease. A comprehensive set of VTE ICD-10 codes were chosen to reduce potential bias introduced by site-specific variance in diagnosis documentation, which may cause decreased specificity in their relationship with IPCd. Further, delayed documentation of VTE ICD-10 codes may inaccurately represent the timing of the disease development in relation to NCCU admission. Similarly, differences in documentation practices exist for quantitative vs. qualitative measures. For example, IPCd application requires manual documentation which may introduce bias, whereas granular and potentially automated data such as ICP measurements affords less opportunity for bias. Additionally, documentation may not always be associated with performance or care given, and may be difficult to operationalize where EHR records and standard terminology is lacking. However, accurate and complete documentation of EHR-based quality measures will also be incentivized by their more widespread utilization, and deviation from practice guidelines are an expected and useful method of providing context for care documentation. The feasibility demonstrated here should motivate the use of these methods moving forward. Another limitation of this study is that it solely evaluated academic medical centers, which effects its generalizability. Further, data contributed to the study came from three participants in the CHoRUS consortium, and although the members of the consortium reflect a variety of critical care specialties, regions, and sizes, selection biases may be present. Other limitations include that ICPM was not differentiated between bundle and EVD, the indications for which may account for timing differences between SAH and TBI; the relatively short duration of admissions may limit the amount of data available for extraction; and the effects of nursing set-up and staffing and disease severity were not accounted for. These and other considerations for each measure are detailed in Table 3.
Table 3.
Measure Definitions, Method Benefits and Challenges
| Numerator | Denominator | Data Source | Benefits | Challenges | Potential False Positives | Potential False Negatives | |
|---|---|---|---|---|---|---|---|
| Acute ICPM | Number of patients receiving ICPM within 24 hours | Number of patients receiving ICPM | EHR: flowsheet records | Automatable collection, scalable | Lacks eligibility criterion | - | - |
| Manual: screening, chart review | Validated, includes eligibility with clinical decision | Time consuming, same source data | - | Missed screens | |||
| Time-to-ICPM | Time (hours) to ICP initiation | Number of patients receiving ICPM within 24 hours | EHR: flowsheet records | Automatable collection, scalable | Inexact, uses first measurement | Active monitoring without recording (e.g. while in OR) | - |
| Manual: procedure observation | Direct measure | Logistical obstacles | - | - | |||
| GCS Frequency | Number of GCS assessments | Length of NCCU stay | EHR: flowsheet records | Automatable collection, scalable | Doesn’t consider contextual factors | - | Non-documented exams |
| Manual: chart review | Validated | Time consuming, same source data | - | - | |||
| Acute IPCd | Number of eligible patients receiving IPCd within 24 hours (excluded: refused, ambulating, limiting injuries) | Number of eligible patients receiving IPCd | EHR: flowsheet records | Scalable, efficient | Inexact timing, documentation as proxy | Ineligible/ unapplied documented | Missed documentation |
| Manual: clinical interview, chart review | Gold standard, refined eligibility with clinical decision | Time consuming, same source data, prospective only | Inclusion in progress note template | - | |||
| Time-to-IPCd | Time (hours) to IPCd application | Number of eligible patients receiving IPCd within 24 hours | EHR: flowsheet records | Scalable, efficient | Documentation as proxy | Ineligible/unapplied or premature documentation | Missed/delayed documentation |
| Manual: clinician interview, patient observation | Gold standard | Time consuming, same source data, prospective only | - | Missed observations |
Conclusion
In summary, we have demonstrated the feasibility, discriminability, and potential for actionability of specific performance measures and quality indicators extracted at scale from electronic health record data. Future work should focus on expanding the number of measures, examining contextual information to explain variation in performance, and evaluating the independent association between site-specific variation and patient outcomes.
Acknowledgments
This study was funded by A New Brain Medical Record for Precision Management of TBI [DoD W81XWH-18-DMRDP-PTCRA].
Appendix A: Selected Flowsheet Measure Names
Glasgow Coma Scale
R ED CLINICAL CALCULATOR - BEST MOTOR RESPONSE (PGCS)
G ED CLINICAL CALCULATOR - GLASGOW COMA SCALE
R ED CLINICAL CALCULATOR - GLASGOW COMA SCALE SCORE
R CPN GLASGOW COMA SCALE BEST MOTOR RESPONSE
G PHS THERAPY GLASGOW COMA SCALE
R PHS IP THERAPY GLASGOW COMA MOTOR RESPONSE
R PHS IP THERAPY GLASGOW COMA SCORE
R GLASGOW COMA SCALE
R CPN GLASGOW COMA SCALE SCORE
R CPN GLASGOW COMA SCALE BEST MOTOR RESPONSE
R CPN GLASGOW COMA SCALE SCORE
R GCS - MOTOR INITIAL HOSPITAL
R GCS - MOTOR PREHOSPITAL
R GCS - TOTAL PREHOSPITAL
R GCS - SCORE INITIAL HOSPITAL
R ED PRE_ARRIVAL GCS SCORE
Intracranial Pressure
R ICP MEAN 1
R AN ICP MEAN
R ICP MEAN 2
R ICP MEAN
R ICP DRAIN STATUS
R ICP MEAN #2
R ICP MONITORING
Mechanical Prophylaxis
R PHS IP PVS INTERVENTIONS
G PVS INTERVENTIONS
R PHS IP GRADUATED COMPRESSION STOCKINGS
G COMPRESSION STOCKINGS
R ACE WRAP
R PHS IP TUBULAR UNI GRIP
R PNEUMATIC COMPRESSORS INITIATED?
R PHS IP OT VASOPNEUMATIC DEVICES DETAILS
R PHS IP PT VASOPNEUMATIC DEVICES DETAILS
R PHS IP UNNA BOOTS
R PHS IP SCD
R INTERVENTION SCD
R PHS IP SCD APPLIED
R IP COMPRESSION BOOTS
R PVS (WDL)
R RLE DVT PROPHYLAXIS
Appendix B: ICD-10 Dictionary
INCLUDE in category if any instance during encounter unless otherwise stated
1. Cerebral ischemia
Cerebral ischemia I67.82
TIA G45. 9, Z86.73
Cerebral infarction I63.0, I63.1, I63.2, I63.3, I63.4, I63.5, I63.8, I63.9, I69.3, G46.3
Carotid occlusion I65.2
Stroke/other vascular occlusions (Retinal) H34.8392, I65.0, I65.1, I65.8, I65.9, I66
MELAS E88.41
Carotid dissection I77.71
Cerebral venous sinus thrombosis I82, I63.6, I67.6, Z86.718
Intracranial and intraspinal phlebitis and thrombophlebitis G08
Other transient cerebral ischemic attacks and related syndromes G45.8
Moyamoya disease I67.5
2. Nontraumatic subarachnoid hemorrhage
Non traumatic Subarachnoid hemorrhage (aSAH) I60, I69.0
Cerebral aneurysm, non-ruptured I67.1
Arteriovenous malformations Q28.1, Q28.2, Q28.3
Carotid aneurysms I72.0
Arteriovenous fistula, acquired I77.0
Stricture of artery I77.1
Compression of vein I87.1
Arteriovenous malformation, other site Q27.39
Arteriovenous malformation, other site Q27.39
Other specified congenital malformations of circulatory system Q28.8
3. Spontaneous Intracerebral hemorrhage
Nontraumatic intracerebral hemorrhage I61, I69.10, I69.159
Nontraumatic intracranial hemorrhage unspecified I62.9
4. Traumatic Brain Injury and Nontraumatic Subdural Hematoma
Epidural hemorrhage/hematoma S06.4X
Subdural hemorrhage I62.00, S06.5X0A
Traumatic subdural hemorrhage S06.5X
Traumatic brain injury S06.2, S06.2X, S06.3, S06.2X0A, S06.2X1A, S06.2X9A, S06.300A
Traumatic subarachnoid hemorrhage (tSAH) S06.6, S06.6X, S06.6X0A
Non traumatic subdural hematoma I62.01
Maxillofacial injury S02.2XXA, S02.401A, S02.92XA, S05.10XA, S05.90XA, S06.9X0A, S06.9X0A, S09.90XA
Only if no other diagnosis 1–13:
Fall W19.
5. Structural or Dynamic CSF Disorders
Hydrocephalus G91.0, G91.1, G91.2, G91.3, G91.4, G91.5, G91.6, G91.7, G91.8, G91.9, Q07
Shunt malfunction T85.02, T85.09
Ventriculitis T85.02
Shunt infection T85.730
Congenital malformations, others Q01, Q04
Cerebrospinal fluid leak G96.0
Other acquired deformity of head M95.2
Arnold-Chiari syndrome without spina bifida or hydrocephalus Q07.00
6. Malignant Brain Tumors
Glioblastoma Multiforme C71.9
Primary Brain Tumor/Malignant neoplasm of brain C71
CNS lymphoma C85.89
Secondary malignant neoplasm of brain C79.31
Metastatic brain tumors C79.31
Hemangioma of other sites D18.09
7. Other head and neck tumors including benign brain/spine tumors
Benign neoplasm of brain, spinal cord D33.0, D33.1, D33.2, D33.4, D33.7, D33.9
Benign neoplasm of cranial nerves D33.3
Benign neoplasm of meninges, unspecified; Meningioma D32.0
Neoplasm of unspecified behavior of brain D49.6
Vertebral Tumors-CNS C41.2
Pituitary tumors and disorders C75.1, D35.2, E22.0, D44.3, E24.0, E23.6
Trigeminal Neuralgia G50.0
Cranial Nerve Disorders, including ophthalmoplegias G52.7, G52.9, H53.2, H54.7
Neoplasm of uncertain behavior of brain and central nervous system D43.0, D43.1, D43.2
Malignant neoplasm of frontal sinus C31.2
Malignant neoplasm of accessory sinus, unspecified C31.9
Unspecified malignant neoplasm of skin of scalp and neck C44.40
Squamous cell carcinoma of skin of scalp and neck C44.42
Malignant neoplasm of head, face and neck C76.0
Benign neoplasm of bones of skull and face D16.4
Benign neoplasm of meninges, unspecified D32.9
Benign neoplasm, unspecified site D36.9
Neoplasm of uncertain behavior of connective and other soft tissue D48.1
Neoplasm of unspecified behavior of respiratory system D49.1
Neoplasm of unspecified behavior of bone, soft tissue, and skin D49.2
Atypical facial pain G50.1
Clonic hemifacial spasm G51.3
Clonic hemifacial spasm, left G51.32
Clonic hemifacial spasm, unspecified G51.39
Other disorders of facial nerve G51.8
Disorders of glossopharyngeal nerve G52.1
Cerebral cysts G93.0
Epidermal cyst L72.0
Other disorders of meninges, not elsewhere classified G96.19
Right temporomandibular joint disorder, unspecified M26.601
Adhesions and ankylosis of right temporomandibular joint M26.611
Adhesions and ankylosis of temporomandibular joint, unspecified side M26.619
8. Seizure disorders (where no other 1–13)
Unspecified convulsions R56.9
Status epilepticus, Status disorder G40
9. Inflammatory, infectious and auto-immune cerebral disorders
Meningitis G00, G01, G02, G03, A87, A17.0, A27.81, A50.41, A51.41, A52.13, A54.81
Meningoencephalitis A17.82, A32.1, B58.2, B60.11, B57.42, B40.81, G04.2
Encephalitis A39.81, A42.82, A85, A86, A92.31, B01.1, B02.0, B05.0, B06.01, B10.0, B94.1, G04.0, G04.8, G04.9, G05.3
Paraneoplastic G13.0, L10.81
Cerebral abscess G06.0, G06.2
HSV encephalitis B00.4
Tb meningitis A17.0
HIV encephalopathy B20
Creutzfeldt-Jacob disease A81.0
Acute Disseminated Encephalomyelitis G04.0
Other specified demyelinating diseases of central nervous system G37.8
Multiple sclerosis G35
Other Encephalopathy G93.40, G93.49, G92, I67.4
Infection following a procedure, initial encounter T81.4XXA
Infection following a procedure, subsequent encounter T81.4XXD
10. Cardiac Arrest, anaphylaxis and risk of anaphylaxis
Cardiac arrest, hypoxic-ischemic encephalopathy, anoxic encephalopathy, anoxic brain injury P91.6, G93.1, I46.2
Cardiac arrest, cause unspecified I46.9
Encounter for desensitization to allergens Z51.6
Urticaria, unspecified L50.9
Adverse effect of other drugs, medicaments and biological substances, initial encounter T50.995A
11. Neuromuscular Disorders
Guillain-Barre syndrome G61.0
Acute demyelinating polyneuropathy G37.9
Chronic Inflammatory demyelinating polyneuropathy G61.81
Amyotrophic Lateral Sclerosis, Primary Lateral Sclerosis G12.21
Myasthenia gravis G70.0, P94.0, G73.3
Other Neuromuscular dysfunction N31.9, G12.20
Botulism A48, A05.1
Chronic inflammatory demyelinating polyneuritis G61.81
12. Spinal cord injury, tumor, and infections
Spinal epidural abscess G06.2
Spinal tumor C72
Cauda equina syndrome G83.4
Spinal fracture S32.0, S12.9XXA
Spinal cord injury S14
Intraspinal abscess G06.1
Disc infection M50.80, M50.90, M46.20, m46.45, m51,84, m51.85,
Neck pain M54.2
Disc disease, compression fracture M50.30, M50.20, M50.00, M48.50XA, M80.08XA, M84.48XA, M84.68XA, M96.1, m48.9
Vertebral dislocation S13.101A
Cervical fracture S12.000A, S12.001A, S12.100A, S12.101A, S12.200A, S12.201A, S12.300A, S12.301A, S14.101A, S14.102A, S14.103A, S14.104A, S12.400A, S12.401A, S12.500A, S12.501A, S12.600A, S12.601A, S14.105A, S14.106A, S14.107A
Sprain of ligaments, Other joint derangement unspecified M24.8, S13.4XXA, S13.8XXA
Spinal stenosis and cervical radiculopathy M54.12, M54.13, M48.02, M47.12
Thoracic disc herniation, dislocation, and radiculopathy, joint pain M54.14, M54.15, M54.16, M54.17, M54.6, M51.34, M51,35, M51.24, M51.25, M94.0, M54.6, S23.101A
Thoracic fracture S22.009A
Thoracic myelopathy M51.04, M51.05
Spondylolisthesis, lumbar region M43.16
Spondylosis without myelopathy or radiculopathy, cervical region M47.812
Spinal stenosis, lumbar region M48.06
Lumbago with sciatica, unspecified side M54.40
Benign neoplasm of spinal meninges D32.1
Benign neoplasm of peripheral nerves and autonomic nervous system, unspecified D36.10
Quadriplegia, unspecified G82.50
Quadriplegia, C1-C4 complete G82.51
Unspecified injury of neck, initial encounter S19.9XXA
13. Vasculopathy
PRES/RCVS I67.83, I67.841
Hypertensive encephalopathy I67.4
Vasculitis L95.0, I73.1, M31.7
Angiitis M31.0
Cerebral venous sinus thrombosis I67.6
Eclampsia O14.95
Footnotes
Conflicts of interest: Dr. Amorim has received research grants from NIH (1K23NS119794), Hellman Fellows Fund, Regents of the University of California (Resource Allocation Program), CURE Epilepsy Foundation (Taking Flight Award), Weil-Society of Critical Care Medicine Research Grant, Zoll Foundation Grant, and American Heart Association (20CDA35310297). Dr. Vespa has received consulting fees from Ceribell, Inc. and UCB, Inc. and has received research grants from NIH (3U54NS100064, 1R01NS113541). Dr. Moorman owns stock in Medical Predictive Science Corporation, has received consulting fees from Nihon Kohden Digital Health Solutions, and has received a research grant from NIH (5R01HD072071). Dr. Hu has received research grants from NIH (5R01NS106905, 1R01NS113541). Dr. Clermont owns stock in NOMA AI, Inc., receives royalties from UpToDate, Inc., and has received research grants from DoD (W81XWH-17-R-BAA180061) and NIH (R01NR013912, R21NS115174, R01DK131586). Dr. Park has received a research grant from NIH (1R21NS113055). Dr. Kamaleswaran has received a research grant from NIH (1R01GM139967). Dr. Foreman has received consulting fees from UCB, Inc. and has received research grants from DoD (W81XWH-18-DMRDP-PTCRA) and NIH (5K23NS101123). Dr. Rosenthal has received consulting fees from UCB, Inc. and Ceribell, Inc. and has received research grants from DoD (W81XWH-18-DMRDP-PTCRA) and NIH (5R01NS117904, 5K23NS105950, 1R01NS113541, 3U54NS100064). The remaining authors declare that they do not have any potential conflicts of interest.
The STROBE observational study checklist was used and is included with this manuscript submission.
Details
This manuscript complies with all instructions to authors.
Authorship requirements have been met and the final manuscript was approved by all authors.
This manuscript has not been published elsewhere and is not under consideration by another journal.
The reported research adheres to ethical guidelines and was conducted under approved investigational review board protocols. This is a retrospective study with de-identified data and was exempt informed consent.
References
- 1.Moheet AM, Livesay SL. Quality improvement in neurocritical care: current state and looking to the future. Current opinion in critical care 2020;26:97–102. [DOI] [PubMed] [Google Scholar]
- 2.Huijben JA, Wiegers EJ, de Keizer NF, et al. Development of a quality indicator set to measure and improve quality of ICU care for patients with traumatic brain injury. Critical Care 2019;23:1–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Huijben JA, Wiegers EJ, Ercole A, et al. Quality indicators for patients with traumatic brain injury in European intensive care units: a CENTER-TBI study. Critical Care 2020;24:1–12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Rhodes A, Moreno RP, Azoulay E, et al. Prospectively defined indicators to improve the safety and quality of care for critically ill patients: a report from the Task Force on Safety and Quality of the European Society of Intensive Care Medicine (ESICM). Intensive care medicine 2012;38:598–605. [DOI] [PubMed] [Google Scholar]
- 5.Hong Y, LaBresh KA. Overview of the American Heart Association “Get with the Guidelines” programs: coronary heart disease, stroke, and heart failure. Critical pathways in cardiology 2006;5:179–86. [DOI] [PubMed] [Google Scholar]
- 6.Howard G, Schwamm LH, Donnelly JP, et al. Participation in Get With The Guidelines–Stroke and its association with quality of care for stroke. JAMA neurology 2018;75:1331–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Schwamm L, Fonarow G, Reeves M, Pan W, Frankel M, Smith E. Get with the guidelines-stroke is associated with sustained improvement in care for patients hospitalized with acute stroke or transient ischemic attack. Circulation 119: 107Á15. 2009. [DOI] [PubMed] [Google Scholar]
- 8.Borg J, Carroll L. Critical evaluation of the existing guidelines on mild traumatic brain injury. J Rehabil Med 2004;43:106–12. [DOI] [PubMed] [Google Scholar]
- 9.de Winkel J, van der Jagt M, Lingsma HF, et al. International Practice Variability in Treatment of Aneurysmal Subarachnoid Hemorrhage. Journal of clinical medicine 2021;10:762. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Livesay S, Fried H, Gagnon D, et al. Clinical performance measures for neurocritical care: a statement for healthcare professionals from the Neurocritical Care Society. Neurocritical care 2020;32:5–79. [DOI] [PubMed] [Google Scholar]
- 11.Sultan AA, West J, Stephansson O, et al. Defining venous thromboembolism and measuring its incidence using Swedish health registries: a nationwide pregnancy cohort study. BMJ open 2015;5:e008864. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Messé SR, Khatri P, Reeves MJ, et al. Why are acute ischemic stroke patients not receiving IV tPA?: Results from a national registry. Neurology 2016;87:1565–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Schwamm LH, Ali SF, Reeves MJ, et al. Temporal trends in patient characteristics and treatment with intravenous thrombolysis among acute ischemic stroke patients at Get With the Guidelines–Stroke hospitals. Circulation: Cardiovascular Quality and Outcomes 2013;6:543–9. [DOI] [PubMed] [Google Scholar]
- 14.Bravata DM, Myers LJ, Cheng E, et al. Development and validation of electronic quality measures to assess care for patients with transient ischemic attack and minor ischemic stroke. Circulation: Cardiovascular Quality and Outcomes 2017;10:e003157. [DOI] [PubMed] [Google Scholar]
- 15.Chan KS, Fowles JB, Weiner JP. Electronic health records and the reliability and validity of quality measures: a review of the literature. Medical Care Research and Review 2010;67:503–27. [DOI] [PubMed] [Google Scholar]
- 16.East AT, Wakefield TW. What is the optimal duration of treatment for DVT? An update on evidence-based medicine of treatment for DVT. Seminars in vascular surgery; 2010: Elsevier. p. 182–91. [DOI] [PubMed] [Google Scholar]
- 17.Haut ER, Noll K, Efron DT, et al. Can increased incidence of deep vein thrombosis (DVT) be used as a marker of quality of care in the absence of standardized screening? The potential effect of surveillance bias on reported DVT rates after trauma. Journal of Trauma and Acute Care Surgery 2007;63:1132–7. [DOI] [PubMed] [Google Scholar]
- 18.Futoma J, Simons M, Doshi-Velez F, Kamaleswaran R. Generalization in clinical prediction models: the blessing and curse of measurement indicator variables. Critical Care Explorations 2021;3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Holder AL, Shashikumar SP, Wardi G, Buchman TG, Nemati S. A Locally Optimized Data-Driven Tool to Predict Sepsis-Associated Vasopressor Use in the ICU. Critical Care Medicine 2021;49:e1196–e205. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Flinn WR, Sandager GP, Silva MB, Benjamin ME, Cerullo LJ, Taylor M. Prospective surveillance for perioperative venous thrombosis: experience in 2643 patients. Archives of surgery 1996;131:472–80. [DOI] [PubMed] [Google Scholar]