Quantifying Mobility in the ICU: Comparison of Electronic Health Record Documentation and Accelerometer-Based Sensors to Clinician-Annotated Video

Supplemental Digital Content is available in the text.

Objectives:

To compare the accuracy of electronic health record clinician documentation and accelerometer-based sensors with a gold standard dataset derived from clinician-annotated video to quantify early mobility activities in adult ICU patients.

Design:

Prospective, observational study.

Setting:

Medical ICU at an academic hospital.

Patients:

Adult ICU patients (n = 30) were each continuously monitored over a median of 24.4 hours, yielding 711.5 hours of video, electronic health record, and sensor data.

Interventions:

None.

Measurements and Main Results:

Electronic health record documentation estimated ambulation (intraclass correlation coefficient, 0.89; 95% CI, 0.78–0.95), sitting out-of-bed (intraclass correlation coefficient, 0.85; 95% CI, 0.72–0.93), and turning events (intraclass correlation coefficient, 0.87; 95% CI, 0.75–0.94) with excellent agreement but underestimated the number of standing, transferring, and pregait activities performed per patient. The accelerometer-based sensor had excellent agreement with video annotation for estimating duration of time spent supine (intraclass correlation coefficient, 0.99; CI, 0.97–0.99) and sitting/standing upright (intraclass correlation coefficient, 0.92; CI, 0.82–0.96) but overestimated ambulation time.

Conclusions:

Our results show that electronic health record documentation and sensor-based technologies accurately capture distinct but complimentary metrics for ICU mobility measurement. Innovations in artifact detection, standardization of clinically relevant mobility definitions, and electronic health record documentation enhancements may enable further use of these technologies to drive critical care research and technology leveraged data-driven ICU models of care.

Despite clinical benefits of ICU early mobility observed in initial studies (1), uncertainties remain regarding the optimal dose of mobility, and identification of patient subgroups most likely to benefit from these interventions. In this regard, at least one randomized controlled trial (RCT) of early mobility versus usual care for patients admitted with acute stroke showed inferior functional outcomes among patients in the intervention group (2). In addition, multiple recent RCTs have yielded nonsignificant results (3–5), raising important questions about the standardization of research methods, timing of mobility interventions, and ability to estimate an effective dose of mobility with which to assess for dose-response relationships.

A major barrier to advancing early mobility research and the development of data-driven models of care is lack of valid, detailed, and reliable methods to measure patient activity in the ICU, analogous to measurement of blood pressure or fluid balance. Routine methods used in outpatient and other inpatient settings are generally not practical or informative to monitor and quantify ICU mobility (6). Although the World Health Organization suggests measuring activity by its four main dimensions: type, duration, frequency, and intensity (7), few technologies and assessment methods allow for comprehensive evaluation of all four components. To measure activity in research, direct methods are preferred for accuracy and limiting recall bias, but require more complex measurement systems that may reduce practicality in clinical settings. Video observation is one such form of activity monitoring that is increasingly used in clinical research (8) and offers additional advantages in activity measurement through the generation of ground truth data upon which activity metrics such as type, frequency, and duration can be derived. Although advances in machine learning will likely enable future automation of video annotation, widespread use of videography for quantification of mobility is not practical at present given the labor-intensive nature of manual review.

Quantification of mobility using wearable activity monitors embedded with accelerometers, pedometers, and gyroscope sensors is also increasingly common in healthcare, albeit with limited evaluation to date in ICU settings (9). Sensor-based measurement methods offer distinct advantages over observation including objectivity, unobtrusiveness, and the ability to make continuous measurements independent of time of day, clinician presence, or patient location. Although accelerometer-based sensors have been validated in healthy ambulatory populations and select acute-care inpatient populations for measuring activity (10), accurate measurement of activity among low mobility populations can be challenging (11), with research groups demonstrating that sensors can overestimate low levels of activity (12) and have difficulty measuring step counts with slower walkers (13).

Existing methods to measure mobility in clinical practice rely on clinician assessments that are manually entered into the electronic health record (EHR). Mobility may be documented in the EHR by several clinicians using multiple data types including unstructured and semistructured text or by entry of structured information from dropdown menus in flowsheets. Although secondary use of EHR mobility-related information is dependent on accurate documentation (14), we could locate no previous studies evaluating the validity of EHR-derived mobility data. Studies evaluating EHR data quality in other contexts have found inaccuracies related to medication lists (15) and medical treatments delivered (16, 17), highlighting the need for additional studies to explore the potential limitations of EHR-derived mobility data for research and clinical decision support.

Valid methods are needed to quantify the time patients spend participating in mobility protocol-driven interventions and engaging in activity on their own, ideally capturing the type, frequency, duration, and intensity of activity performed. Greater understanding of the advantages and limitations of emerging technologies to quantify patient movement in the ICU will improve mobility measurement for research and enable expansion of data-driven care models. The purpose of this study was to compare the accuracy of EHR clinician documentation and accelerometer-based sensors to quantify ICU early mobility activities compared with a gold standard dataset derived from clinician-annotated video recordings.

METHODS

We conducted a prospective observational study of adults hospitalized in the medical ICU (MICU) of an academic medical center in California following Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines (Supplemental Table 1, Supplemental Digital Content 1,http://links.lww.com/CCX/A153). An early mobility initiative was launched in the MICU in 2012 and became standard of care across all adult ICUs in 2016. Study protocols were approved by the University’s Institutional Review Board, and study equipment was evaluated by our Health Information Technology department prior to deployment.

Participants

Adult MICU patients eligible for early mobility interventions were recruited between May 2017 and January 2018. Inclusion and exclusion criteria for study participation are detailed in Table 1. We identified patients eligible for early mobility interventions and participation in the study through EHR screening and discussions with the interdisciplinary team. In addition to the study-specific criteria in Table 1, clinical eligibility for ICU early mobility was routinely assessed each day in the MICU according to American Association for Critical Care Nurses (AACN) safety screening guidelines for cardiac stability, oxygenation and ventilation requirements, vasopressor use, level of consciousness, and neurologic stability (18). As only one to two participants could be enrolled simultaneously due to limited availability of video equipment, we used a random number generator in the Python programming language to randomly select patients to approach for enrollment when there were more than two patients eligible to participate in the study on a given day. Informed consent was obtained from patients and/or surrogates when consent was not possible due to sedation or altered mental status.

TABLE 1.

Selection Criteria for Study Participation

The primary objective of the study was to compare EHR and accelerometer-based activity estimates against video observation. We used a paired sample t test to determine the number of subjects needed to discriminate between mean activity frequency counts for each measurement method. We estimated that a sample size of 32 subjects would provide 80% power to detect a 0.5 sd difference in activity frequency between the gold standard and experimental ICU activity measurement methods, using an approach employed previously in circumstances where insufficient literature exists to estimate an expected effect size (19).

Materials

A security camera system with night vision was used for continuous video recording of patient movement. Multiple precautions were taken to preserve patient privacy and ensure the security and confidentiality of data acquired through video observation. After mounting the system in the patient’s room, the camera was attached to a power source, and the system was physically secured. To minimize unnecessary data acquisition, no audio was obtained. Camera angle was setup to minimize capture of facial images of staff or other visitors on video. All video data were stored on password protected, university computers in limited-access directories. A Certificate of Confidentiality was also obtained from the National Institutes of Health to protect the privacy of participants and nonsubjects inadvertently recorded on video.

We used a commercially available, clinical-grade three-axis accelerometer designed to monitor mobility among hospitalized adults (20) (Leaf Healthcare, Pleasanton, CA). The U.S. Food and Drug Administration-approved device uses a 1.5-inch disposable sensor that attaches to a patient over the sternum. Although the Leaf sensor may be used in patients with pacemakers, the manufacturer recommends placement at least 10-cm away from a pacemaker. As a result, we excluded patients with pacemakers to ensure that the same location on the sternal chest could be used for all subjects. The sensor’s proprietary configuration recorded samples at least once every 10 seconds and sent data through a dedicated local wireless network to a study server. At the time of the study, no reliability or validity data on this device had been published for quantifying ICU mobility.

The University Health System’s EHR system (Epic, Verona, WI) has been used by the medical center since 2008, where nurses, physical therapists, and physicians document mobility-related activities in a number of ways, using both flowsheets and narrative free-text notes.

Procedures

After enrollment, the ICU room was equipped with a video camera system, and a sensor was placed on the participant and activated to initiate data transmission. Participants then underwent 24 hours of continuous activity monitoring to ensure representative sampling of day and night mobility interventions and associated documentation practices. After approximately 24 hours of data were collected, equipment was removed from the ICU room, and video data stored on a digital video recorder was transferred to a secure University server for annotation. Activity reports generated by the sensor system’s proprietary software were provided by Leaf Healthcare Inc. All data compiled during the enrollment period were entered and stored using Research Electronic Data Capture (REDCap) (21).

Twenty-four hours after the activity monitoring period concluded, we conducted a retrospective review of patient characteristics, health status during enrollment, and mobility data from the EHR using REDCap. Flowsheet fields and progress notes recorded by clinicians were examined to extract the frequency of early mobility events that occurred during the identical time period as video and accelerometer monitoring. Activities that were observed in both a flowsheet and a progress note during the same time period were assumed to be duplicate entries. Transfers occurring between recorded activities were not assumed unless explicitly documented. A second reviewer abstracted 23% of participant records to verify the mobility types and event frequencies recorded. All EHR data abstraction was performed at least 2 months prior to video annotation to allow for a washout period between analyses.

Data Analysis

Definitions for activity were categorized using the four-phase quantification approach of ICU early mobility and progressive exercise clinical guidelines from the AACN and the Society of Critical Care Medicine (SCCM) (22) (Supplemental Fig. 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A153). Video recordings were used as the criterion standard to measure mobility, similar to previous validation studies (23). We developed a coding scheme to annotate ICU mobility observed in the video recordings using an iterative process of literature review, convening with multidisciplinary experts, and testing codes using recordings with simulated patients (24). The final coding scheme consisted of recording all patient movement lasting greater than 10 seconds in an Excel template (Supplemental Fig. 2, Supplemental Digital Content 1, http://links.lww.com/CCX/A153). The first author annotated all videos and another author independently reviewed 5% of video files, representing 35 hours of video, stratified across mobility phases (Supplemental Fig. 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A153).

In addition to classifying ICU mobility types, we also quantified mobility according to frequency and duration. In the EHR, only mobility type and event frequency could be consistently extracted, although all AACN and SCCM activity types were found in the EHR and compared with video annotation (Supplemental Fig. 3, Supplemental Digital Content 1, http://links.lww.com/CCX/A153). At the time of data collection, only activity duration for time spent lying, sitting/standing upright, and ambulation captured through the sensor could be compared with video annotated activities. A post hoc secondary analysis was also conducted to compare accelerometer-derived ambulation time to video under the following conditions: (1) use of a broader ambulation definition to include all standing mobility activities (e.g. all activity while patient stands including ambulation, transitions involving standing, range of motion while standing, and marching/sidestepping) and (2) removal of participant cases who received high-frequency chest wall percussion therapy as part of their treatment due to a known potential for ambulation artifacts with this therapy by the sensor manufacturer.

Quantitative variables were summarized with means and 95% CIs or medians with interquartile ranges depending on normality of the data assessed using the Shapiro–Wilk test. Categorical variables were summarized with proportions. Intraclass correlation coefficients (ICCs) were calculated to compare agreement between methods on a per-patient basis (25). We also conducted Wilcoxon signed-rank tests to compare the difference between mobility estimates and determine if the EHR and sensor were biased toward over or underestimation of activity frequencies and durations. Bland–Altman plots were used to display agreement, by plotting the difference of the two measurements against the mean of the two measurements for each patient (26). Cohen’s Kappa was calculated to estimate interrater agreement between reviewers of both video and EHR data (27). All hypotheses were two-sided and tested at a significance level of 0.05. Statistical analyses were performed using Stata, version-13 software (28).

RESULTS

A total of 711.5 hours of video, sensor, and EHR data were recorded, reviewed, and analyzed across the cohort (median of 24.4 hr of data per patient; interquartile range, 23.5–25.7). Corruption of video data in one participant and loss of a sensor in another resulted in a final cohort size of 30 participants (Fig. 1). Participant characteristics can be found in Table 2; characteristics related to sensor use are presented in Supplemental Table 2 (Supplemental Digital Content 1, http://links.lww.com/CCX/A153). Across the entire cohort, there were seven ambulation events (1%), 131 standing events (18%), 224 sitting events (30%), 58 out-of-bed sitting events (58%), 249 turning events (34%), and 72 range of motion exercise events (10%) observed through video. Total time spent performing mobility activities for all patients included 0.4 hours ambulating (<1%), 2.9 hours standing (<1%), 74.2 hours sitting (10%), and 634.0 hours lying supine (89%). Cohen’s Kappa statistic for the intercoder reliability of 23% of EHR records and 5% of video recordings to verify reviewer-recorded mobility events was κ = 0.90 and κ = 0.93, respectively.

Figure 1.

Consort diagram for participant screening, enrollment, data collection, and analysis. EHR = electronic health record.

TABLE 2.

Demographic and Clinical Characteristics for Participants

Agreement Between EHR Versus Clinician-Annotated Video

EHR documentation demonstrated excellent agreement with video for estimating the number of turning (ICC, 0.87; 95% CI, 0.75–0.94), sitting out-of-bed (ICC, 0.85; 95% CI, 0.72–0.93), and ambulation events (ICC, 0.89; 95% CI, 0.78–0.95) (Table 3). For all other activities, EHR-based estimates of mobility events demonstrated poor agreement, with ICC values between 0.06 and 0.45 (Table 3). Agreement varied among out-of-bed sitting subtypes and locations (Supplemental Table 3, Supplemental Digital Content 1, http://links.lww.com/CCX/A153). When comparing median activity events per patient and directional bias using the Wilcoxon signed-rank test, the EHR significantly underestimated the frequency of turning and repositioning events, total number of sitting events, transferring events, and standing/pregait events compared with video annotation (Table 3). The Bland–Altman analyses provide additional visual representation of agreement between the EHR and video-derived mobility data (Fig. 2). The largest differences between the two methods were for estimating the number of transferring, sitting, and standing/pregait events, where the EHR underestimated the number of events by a mean difference of –5.27, –5.63, and –4.47, respectively. Although our study sample size was small, the occurrence of points outside of the limits of agreement in the Bland–Altman plots suggests there was less agreement between the EHR and video when some mobility types occurred at higher frequencies.

TABLE 3.

Agreement Between Video Annotation, Electronic Health Record Documentation and Accelerometer-Based Actigraphy for Measuring Mobility Episode Frequency and Duration Per Patient (n = 30)

Figure 2.

Visual depiction of agreement in activity frequency between video and electronic health record (EHR) estimations per patient, according to the Bland–Altman method. Each value represents the difference in activity frequency estimates between the two methods (EHR minus Video) against the mean of the two methods. Points above the y-axis zero line indicate overestimation by the EHR, and points below the x-axis zero line indicate underestimation. The dotted lines represent the upper and lower limits of agreement (LoA) (± 1.96 sd).

Agreement Between Accelerometer Versus Clinician-Annotated Video

Examination of the accelerometer-based sensors for measuring mobility type and duration demonstrated excellent agreement for estimating time spent supine (ICC, 0.99; 95% CI, 0.97–0.99) and upright (ICC, 0.92; 95% CI, 0.82–0.96), but poor agreement for ambulation time (ICC, 0.01; 95% CI, 0.00–0.36), with the sensor significantly overestimating the duration of ambulation (Table 3). In our post hoc ambulation analyses, we found that agreement substantially improved between the two methods when we expanded our ambulation definition to “standing activities, including ambulation”, combining standing activities such as marching in place with ambulation and excluded seven patients who received percussion and/or oscillation chest therapy from the cohort (ICC, 0.82; 95% CI, 0.63–0.92) (Supplemental Table 4, Supplemental Digital Content 1, http://links.lww.com/CCX/A153).

DISCUSSION

We conducted an observation study evaluating EHR documentation and accelerometer-based sensors for quantifying ICU early mobility, comparing these two methods to a large, 700-hour gold standard video dataset. We found EHR documentation accurately quantified the frequency of turning, ambulation, and out-of-bed sitting events but underestimated most other types of activity with no ability to consistently measure activity duration. In comparison, the accelerometer demonstrated excellent agreement for measuring duration of time spent supine and sitting/standing upright but overestimated ambulation time. Although neither method accurately measured all aspects of ICU early mobility, together, the two technologies provided complementary quantitative data on three of four World Health Organization domains of activity assessment (type, frequency, and duration) (7).

Our study illustrates important barriers to the validity of EHR-derived mobility data and highlights challenges for reuse downstream. With its widespread use and availability, EHR-derived mobility data are the standards for clinical monitoring of ICU mobility; however, concerns related to recording bias and data heterogeneity potentially limit its use. Among the seven ICU early mobility activity types and six activity subtypes we examined, eight activity types/subtypes were captured with poor agreement, and the frequency of eight activities was significantly underestimated by EHR documentation (Supplemental Table 3, Supplemental Digital Content 1, http://links.lww.com/CCX/A153). Researchers have also found bias toward underrepresentation of other ICU healthcare interventions when comparing EHR documentation with ground truth (17, 29). Even if bias from EHR-derived data is accounted for, substantial heterogeneity in documentation will make automation of ICU mobility data for secondary use challenging. To estimate EHR-recorded activity, we searched 17 structured data fields across two flowsheet templates and six note types from three different clinician groups to capture all the potential locations and methods mobility might be recorded. The flowsheet fields examined showed inconsistent recording of activity duration, frequent use of free-text comment entry associated with a discrete flowsheet field, and the presence of multiple fields to describe similar assessments or interventions. Our findings are consistent with prior EHR data quality studies, highlighting the need to streamline EHR data entry to decrease workarounds and clinician documentation burden (30, 31) and to support the assertion that bias and heterogeneity need to be addressed before quantitative EHR-derived mobility data can be reliably used for longitudinal research, quality improvement, and clinical decision support (32).

Accelerometer-quantified physical activity offers an alternative solution for standardized collection, documentation, and integration of mobility data into the EHR. Accelerometer-based sensors have demonstrated utility in a number of healthcare applications, mostly in outpatient settings (33, 34). Despite several benefits, accelerometers have specific challenges, including application of sensors and algorithms tuned on one population and applied to another, standardization of definitions used to classify activity across devices, and development of clinical artifact recognition algorithms to improve accuracy. In our study, accelerometer-based estimation of ICU patient supine and upright time demonstrated excellent agreement, however ambulation time performed as well only after a number of conditions were applied during secondary analysis pertaining to artifact removal and activity definitions. Similarly, research groups have found mixed results when attempting to accurately quantify ICU mobility using activity sensors. Excellent agreement has been found using accelerometers to estimate standing/walking time among adult inpatients (35) and sit-stand transitions among ICU survivors (36); however, ankle versus wrist placement in another MICU study conflated activity and sleeping time (37), and two 2005 studies found accelerometers overestimated ICU activities (12, 38), which we also observed with ambulation. Recent motion sensor systems have had success accurately estimating ICU activity using a collapsed version of the ICU Mobility Scale (39) and detecting out-of-bed transitions (40); however, results from one system showed lower levels of accuracy when estimating higher levels of ICU activity (41).

We also found that agreement between the sensor and our gold-standard changed substantially depending on how mobility was defined. Activity categorizations used in sensor-based monitoring may differ from mobility measurement strategies important in clinical practice, where manufacturers may have configured sensors in ways that may not generalize equally to all patient populations or clinical settings. Many accelerometers provide raw activity counts dependent on conversion into clinically meaningful thresholds (12, 38) or intensity levels recommended for ambulatory populations (42) that may not be relevant to inpatient populations, particularly those with critical illness. We used an a priori ambulation definition based on ICU guidelines, whereas in our secondary analysis, we used a combined definition of standing/ambulation more germane to prior activity monitoring studies (10, 35). With multiple devices coming to market, the potential for differences in quantification, and the increased secondary use of data for research, clinical operations, and care planning, understanding the nuances and bias in the accuracy of these data types are necessary.

Our finding that neither measurement modality was able to individually capture all activity types and measurement dimensions suggests that a combined approach using both clinician documentation and sensor data may improve the accuracy and comprehensiveness of ICU mobility monitoring for practice and research. Technologies such as video observation and accelerometer-based sensors could be used in future trials to objectively document the frequency, type, and duration of early mobility and related activities in the ICU. For example, video monitoring could enhance quantification of early mobility activity subtypes, frequency, and duration, whereas accelerometer-based sensors could improve measurement of activity intensity and duration of bed rest. Recent studies of automated early mobility measurement have expanded the potential range of technologies to also include motion sensors (39, 41) and deep learning-based image analysis (40). Refinement of these and more traditional accelerometer-based technologies will likely continue to improve the utility of sensor-derived data, but each has limitations that may ultimately benefit from complementary clinician- and context-derived information. Although novel technologies will need to be carefully evaluated for accuracy and generalizability, future research will also need to address technical interoperability, usability, and effects on clinician workflows to ensure EHR integration and successful clinical adoption. Ultimately, more comprehensive quantification of ICU early mobility will allow the effective estimation of a “dose” of early mobility that will in turn enable clinicians in carrying out and measuring the real-time effects of ICU early mobility interventions.

Although our study is strengthened by its use of multiple data sources and a robust gold standard dataset to evaluate quantitative mobility in the ICU, a number of limitations should be noted. The small, single-center nature of our study may limit generalizability of our findings to other ICU patients and institutions with different EHR configurations and documentation practices. Although our sample size and monitoring period were larger than similar sensor validation studies (12, 36, 38, 39, 41), 24 hours still provided a limited view of ICU mobility, especially as higher levels of mobility occur infrequently. The study was powered for a sample size of 32 participants; therefore, after loss of data for two subjects (Fig. 1), the smallest difference between methods we could detect was 0.52 sds rather than the target of 0.5. We also conducted the study with only one accelerometer model and body placement. Because we used a commercial sensor, we were limited in sensor configuration and algorithm properties such as sampling rate, activity thresholds, and artifact identification. It is possible we would have observed different activity estimates using a different device, body placement, and configuration settings. Last, our use of a single reviewer to annotate video files and extract EHR data could have biased the video annotation despite our use of a 2-month or more wash out period between EHR data extraction and video annotation.

CONCLUSIONS

We validated two concurrent methods of measuring ICU mobility, comparing each with a gold standard video dataset and found that both were imperfect but complimentary. Although the EHR correctly estimated the frequency of mobility types such as ambulation and out-of-bed sitting events, the accelerometer-based sensors performed with excellent agreement for quantifying time spent supine and sitting/standing upright. Given the multifaceted construct of measuring ICU mobility, an individual measurement tool may not be sufficient to accurately and reliably measure all features, including type, duration, frequency, and intensity of mobility, and instead a combined measurement approach may be more suitable. Future research should focus on EHR mobility documentation workflows to improve the validity and reliability of the recorded data while preserving clinician efficiency, automation of data abstraction, and video analysis through machine learning, and on the development of advanced sensor algorithms capable of detecting the low levels of activity typical of ICU mobility while also removing clinical artifacts that may contribute to inaccuracies. These advances will enable further research into potential dose-response relationships between ICU early mobility interventions and patient outcomes and stimulate development of technology-leveraged, data-driven ICU models of care.

ACKNOWLEDGMENTS

We graciously acknowledge the patients, families, MICU nurses, physical therapists, and physicians who participated in and contributed to this study. We also wish to thank members of the UC Davis Critical Care Informatics Laboratory, IT Health Informatics, Gordon and Betty Moore Foundation, and Dr. Irene Cortés-Puch, for her thoughtful review of and contributions to this article.